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EXECUTIVE 

SUMMARY 


This  report  describes  an  initiative,  called  the  Continuum  of  eLearn¬ 
ing  (CoL),  which  addresses  some  of  the  gaps  and  limitations  of  indi¬ 
vidual  and  collective  training  by  providing  blended-learning  training 
packages  to  better  integrate  online  learning  activities  into  the  Joint 
Exercise  Live  Cycle.  The  CoL  is  a  web-based  training  package,  de¬ 
veloped  by  the  Joint  Training  of  the  Joint  Staff  J7,  and  it  is  intended 
to  represent  the  “next  evolution”  of  training  on  JKO.1 

Gaps  and  Opportunities 

Collective  Training 

Each  year,  the  Joint  Staff  J7,  coordinates  dozens  of  large-scale  joint 
and  coalition  training  events,  and  military  leaders  are  constantly 
looking  for  ways  to  enhance  the  events’  training  outcomes.  In  par¬ 
ticular,  joint  training  personnel  and  the  literature  have  identified  five 
areas  that  could  be  improved  upon: 

1.  There  are  often  significant  numbers  of  “untrained”  staff.  It  is 

not  uncommon  for  staff  members  to  miss  a  collective  training  exer¬ 
cise  because  of  late  assignment,  in  country  duties,  or  other  logistical 
limitations.  These  personnel  remain  “untrained”  since  no  alternative 
mechanisms  currently  substitute  for  a  collective  event.  This  creates 
weaknesses  in  the  shared  knowledge-base  of  operational  staffs. 

2.  Training  and  education  events  lack  blending.  Despite  the  qual¬ 
ity  of  training  offered,  it  is  currently  executed  in  stovepipes,  with 
little  correspondence  among  individual  training  concepts,  joint  aca- 

1  Joint  Knowledge  Online  (JKO)  develops  and  delivers  online  joint  training  to  prepare 
and  assist  individuals  to  support  joint  and  coalition  operations.  JKO  is  the  DoD’s  unique 
and  authoritative  capability  for  providing  operationally  relevant  and  globally  accessible 
training  to  support  individual  joint  preparedness  and  exercises.  Combatant  commands  are 
dependent  on  JKO  as  an  integral  component  of  their  individual  joint  training  plans. 


PANAMAX  20 1 2  participants  complete  CoL 
courses  prior  to  the  start  of  the  collective 
exercise  (July  20 1 2) 


SECTION  SUMMARY 

The  Continuum  of  eLearning 
(CoL)  is  a  methodology  and  ca¬ 
pability  that  blends  online  learn¬ 
ing  with  joint  collective  training. 

CoL  1 .0  was  developed  in  2012 
and  tested  at  PANAMAX. 

Joint  staff  members’  subjective 
reactions  and  objective  learning 
effectiveness  scores  were  signifi¬ 
cantly  higher  in  the  CoL  (experi¬ 
mental)  condition  as  compared 
to  their  previous  experiences  and 
the  control  condition,  respec¬ 
tively. 
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demies,  and  collective  exercise  objectives. 

3.  Personnel  possess  Service-specific  mindsets. 

Personnel  assigned  to  a  joint  billet  may  not  have 
served  in  any  joint  position  before.  Consequently, 
they  may  exhibit  Service-centric  attitudes  that  ini¬ 
tially  inhibit  their  effectiveness  in  their  joint  roles. 

4.  Commanders  and  observer-trainers  (OTs) 
could  make  use  of  more  detailed,  diagnostic 
outcome  data.  Targeted,  objective  assessment 
of  personnel’s  cognitive  capacities  is  rarely  con¬ 
ducted  across  the  entire  cohort  of  trainees.  Hav¬ 
ing  enhanced  individual  readiness  data  would  give 
commanders  more  constructive  insights  into  their 
staff’s  preparedness,  and  observer- trainers  (OTs) 
insight  into  what  aggregate  knowledge  gaps  they 
should  address  during  various  training  activities. 

5.  It  is  unclear  how  much  training  personnel  re¬ 
tain  between  events.  Staffs  are  constantly  chang¬ 
ing,  due  to  routine  rotations  and  re-assignments. 
As  such,  there  is  an  ongoing  struggle  to  maintain 
a  high  “band  of  excellence”  in  the  experience  and 
expertise  of  the  permanent  staff.  In  addition  to  this 
unique  difficulty,  joint  training  personnel  also  face 
the  conventional  challenge  of  training  transfer.  It  is 
unclear  how  much  training  transfers  to  individuals 
during  a  collective  training  event,  as  well  as  how 
much  of  that  transferred  knowledge  personnel  ac¬ 
tually  retain  between  events. 

Online  Learning 

Supplementary  online  learning  seems  like  an  ob¬ 
vious  solution  to  the  challenges  outlined  above. 
However,  despite  the  potential  benefits  of  well- 
designed  e-learning,  online  courses  in  practice  not 
possess  a  sufficient  level  of  rigor  and  this  can  nega¬ 
tively  affect  their  effectiveness,  utility,  and  appeal. 

As  part  of  this  effort’s  front-end  analysis,  we 
conducted  structured  interviews  with  seven  active 
duty  and  government  civilian  stakeholders  associ¬ 
ated  with  the  major  military  e-learning  enterprises 
in  the  Air  Force,  Army,  Navy,  Marine  Corps,  and 
Joint  Staff.  These  subject-matter  experts  offered 
informed  opinions  about  military  online  courses, 


Table  I .  I .  Summary  of  gaps  in  military  collective  training  and  ways 
that  online  and  blended  learning  can  help  address  each  issue. 


Issue 

Resolution  Strategy 

“Untrained”  Staff 

Staff  members  frequently  miss 
the  joint  exercise  before  a 

Joint  Task  Force  forms  (e.g., 
due  to  late  assignment  or  in¬ 
country  duties) 

24/7  e-Learning  Access 

An  obvious  benefit  of 
e-learning  is  that  it  allows  24/7 
access;  so,  staff  can  receive 
some  training,  even  if  they 
miss  the  collective  exercise 

Stovepipe  Events 

Often,  learning  opportunities 
are  executed  in  stovepipes, 
without  explicit  links  to  related 
training  events 

Blended  Learning 

Use  blended  learning 
processes  to  help  ensure  that 
related  learning  opportunities 
are  integrated  together 

Service  Mindsets 

When  first  assigned  to  a 
joint  billet,  Service  members 
may  lack  joint  mindsets, 
which  initially  inhibits  their 
effectiveness  in  joint  roles 

Contextualize  Content 

Use  narrative  and  humanized 
instructional  interventions  to 
make  joint  doctrine,  culture, 
and  vocabulary  more  relatable 
to  online  learners 

Outcome  Data 

Commanders  and  OTs  could 
make  use  of  more  detailed, 
more  diagnostic  training 
outcome  data 

Data  Dashboard 

Use  better,  more  diagnostic 
measures  to  compile 
meaningful  individual  readiness 
data  and  then  translate  this  for 
OTs  and  commanders 

Retention 

Staff  members’  skills  and 
knowledge  may  decay 
between  training  events,  and 
each  Staff  member  will  likely 
require  a  unique  remediation 
plan 

Current  and  Targeted 

Provide  enduring  online 
content,  keep  it  concurrent 
with  real-world  operations 
and  best  practices,  and  tailor 
course  assignments  to  each 
student’s  learning  needs 

and  they  suggested  that  military  e-learning  could 
be  improved  in  areas  such  as  the  sophistication  and 
specificity  of  content,  the  effectiveness  of  learning 
assessments,  students’  motivation  to  use  the  online 
courseware,  courses’  seeming  lack  of  relevance,  and 
courses’  lack  of  usability. 

Continuum  of  eLearning 

The  CoL  was  initiated  to  address  the  gaps  in  joint 
collective  training  while  also  improving  upon  the 
delivery  of  online  learning.  The  CoL  represents 
both  a  capability  (i.e.,  the  instructional  content) 
and  a  methodology  (i.e.,  the  implementation  ap¬ 
proach)  for  bolstering  joint  training.  It  employs 
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research-based  best  practices  for  blended  learn¬ 
ing  to  better  support  adult  education  and  training, 
improve  learning  outcomes,  and  enhance  students’ 
motivation  to  learn  online. 

CoL  Components 

As  stated  above,  the  CoL  represents  both  an  in¬ 
structional  capability  as  well  as  a  systematic  ap¬ 
proach  for  developing  online  courses,  blending 
them  with  small-group  and  collective  training 
events,  and  ensuring  that  instructional  best  prac¬ 
tices  are  continuously  infused.  More  specifically, 
the  CoL  includes: 

•  Self-paced,  e-learning  courses 

•  Small-group  online  simulation 

•  Strategies  for  blending  learning/training 

•  Strategies  for  enhancing  development 

•  Strategies  for  enhancing  execution 

CoL  Version  1 .0 

Version  1.0  of  the  CoL  was  implemented  in  2012 
through  JKO.  To  date,  CoL  courses  include  con¬ 
tent  from  the  100  (i.e.,  “basic”)  and  200  (i.e.,  “in¬ 
termediate”)  levels  of  the  Joint  Force  Command 
curriculum,  as  implemented  in  support  of  Combat¬ 
ant  Command  and  Joint  Task  Force  exercises.  CoL 
VI. 0  also  incorporates  the  following  best  practices 
in  order  to  better  support  student  learning: 

1.  Emphasize  higher- order  learning 

2.  Use  pre-tests  for  performance  adaptation 

3.  Incorporate  higher-order  assessments 

4.  Use  formative  and  summative  assessments 

5.  Employ  mastery  learning  strategies 

6.  Incorporate  historical  vignettes 

7.  Create  higher-levels  of  interactivity 

8.  Ensure  better  alignment 

CoL  VI  .0  Beta  Testing 

Version  1.0  of  the  CoL  was  tested  at  PANAMAX 
2012,  a  US  Southern  Command  multinational 
training  exercise.  Data  collection  for  this  beta  test 
took  place  20  May-17  Aug  2012  in  Mayport, 
Doral  (i.e.,  Miami),  and  Suffolk.  Approximately 


Comparison  of  Previous  Online  Learning 
Experiences  to  CoL 


Efficiency  Relevance  Effectiveness  Satisfaction 


Figure  I.  Comparison  of  previous  online  learning  experiences 
to  participants’  CoL  reactions.  Participants  rated  their  CoL 
experiences  more  positively  than  their  previous  military  e-learning 
experiences. 

200  US  personnel  from  the  Tier  I  or  Tier  II  staffs 
participated.  Half  of  the  participants  (i.e.,  the  ex¬ 
perimental  group)  completed  the  following  online 
CoL  courses  prior  to  the  event: 

•  JFC  100  Joint  Operations  Planning 

•  JFC  100  Interorganizational/Multinational 

•  JFC  200  Interorganizational  Coordination 

Throughout  the  beta  test,  the  research  team 
evaluated  the  efficacy  of  the  CoL  through  a  multi¬ 
part  experiment,  documenting  the  learning  effec¬ 
tiveness  of  the  courses  as  well  as  their  usability,  mo¬ 
tivational  effects,  operational  relevance,  and  ability 
to  engender  a  “joint  mindsetf’Just  over  100  partici¬ 
pants  ( n  =  106)  completed  the  experimental  CoL 
content  and  some  portion  of  the  experimental  bat¬ 
tery.  The  others  ( n  =  90)  participated  as  a  control 
group.  Both  groups  completed  knowledge  tests, 
joint  mindset  questionnaires,  and  reactions  surveys. 

Participant  demographics.  Most  participants  were 
Sailors  (about  42%),  but  Airmen  (27%),  Soldiers 
(25%),  and  Marines  (6%)  were  also  represented. 
Participants  were  nearly  equally  divided  between 
active  and  reserve  components,  and  most  partici¬ 
pants  held  mid-level  staff  ranks  (E6-E7  or  03-04; 
about  59%  of  participants),  although  higher  (30%) 
and  lower  (11%)  ranks  were  also  present. 

Differences  by  rank  and  Service.  Differences 
based  upon  rank  and  Service  were  found  on  the 
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knowledge  questionnaire  and  post-course  reac¬ 
tions  survey.  US  Army  personnel  achieved  higher 
test  scores  as  compared  to  personnel  from  the  other 
Services,  and  as  expected,  higher-ranking  person¬ 
nel  also  earned  better  test  scores.  Lower  ranking 
personnel,  however,  rated  the  online  courses  more 
favorably.  Based  upon  participants’  free-response 
comments,  it  appears  that  higher-ranking  partici¬ 
pants  saw  the  need  for  the  online  courses  but  felt 
like  they  already  had  a  working  knowledge  of  the 
content. 

Previous  online  learning  experiences.  Before 
completing  any  CoL  courses,  participants  were 
asked  about  their  previous  online  learning  expe¬ 
riences.  Overall,  participants  rated  their  pre-CoL 
online  learning  experiences  slightly  below  “neutral” 
on  a  five-point  scale  (M  =  2.87,  SD  =  0.93). 

Reaction  (Kirkpatrick’s  Level  1).  After  taking  the 
three  CoL  courses,  experimental  participants  com¬ 
pleted  surveys  regarding  the  perceived  efficiency, 
relevance,  and  effectiveness  of  the  e-learning  as  well 
as  their  overall  satisfaction  with  the  learning  expe¬ 
rience.  Participants  rated  the  CoL  courses  slightly 
above  “neutral”  on  a  five-point  scale  (M=  3.24, 
SD  =  0.88).  Paired  samples  /-tests  indicated  statis¬ 
tically  significant  differences  between  the  partici¬ 
pants’  previous  online  learning  experiences  and  the 
CoL  online  experience  (see  Figure  1).  Although 
participants’ reactions  to  the  CoL  courses  remained 
lukewarm,  these  courses  were  rated  more  highly 
than  the  online  courses  the  participants  had  previ¬ 
ously  experienced. 

Learning  (Kirkpatrick’s  Level  2).  All  participants 
completed  identical  knowledge  tests.  Outcomes 
from  a  univariate  between-subjects  ANCOVA, 
controlling  for  rank  and  Service,  indicate  that  there 
is  a  statistically  significant  difference  between  the 
control  {n  =  47,  M  =48.83%)  and  experimental 
(n  =  57,  M  =  59.09%)  group  scores,  F(2, 48)  =  8.58, 
/>=.001.  Participants  who  completed  the  CoL 
scored,  on  average,  21%  better  on  the  knowledge 
test  versus  participants  in  the  control  group  (see 
Figure  2). 


Knowledge  Test  Scores  by  Group 
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Figure  2.  Comparison  of  experimental  (CoL)  and  control  group 
participants’  knowledge  scores  reveals  that  the  experimental 
group  scored  significantly  higher  on  learning  outcomes. 

Recommendations 

As  a  result  of  the  CoL  VI. 0  research  and  develop¬ 
ment  initiative,  the  evaluation  team  compiled  the 
following  recommendations  for  enhancing  joint 
individual  and  collective,  training  and  education. 
See  Table  1  for  a  summary. 

1.  Revise  e-learning  specifications  for  blended 
learning.  This  initiative  uncovered  valuable  recom¬ 
mendations  about  how  to  best  design  joint  e-learn- 
ing  and  blended  learning  experiences  in  support  of 
collective  training.  These  recommendations  should 
be  formalized  into  official  instructions  for  relevant 
stakeholders,  such  as  instructional  systems  design¬ 
ers  (ISDs)  and  observer-trainers  (OTs). 

2.  Develop  blended  learning-training  packages. 

To  facilitate  blended  learning,  blended  learning¬ 
training  packages  should  be  developed  that  include 
intentional  linkages,  cross  references,  and  blending 
guidance  for  OTs.  To  fully  support  this  recommen¬ 
dation,  OTs  will  also  require  training  on  how  best 
to  use  the  new  blended  learning  resources. 

3.  Develop  an  automated  data  “dashboard”  to 
support  blended  learning.  We  recommend  the 
development  of  a  data  “dashboard”  that  automati¬ 
cally  analyzes  personnel’s  learning  outcomes  (from 


Experimental  Control 


<<  UNCLASSIFIED>  > 


Continuum  of  eLearning  |  Executive  Summary 


v 


Table  2.  Summary  of  high-level  recommendations  for  the  overall 
continuum  of  joint  learning  and  training 

Recommendations 

1 .  Revise  e-learning  specifications  for  blended  learning 

2.  Develop  blended  learning-training  packages 

3.  Develop  a  blended  learning  data  “dashboard” 

4.  Improve  CoL  assessment  creation  and  validation 

5.  Implement  adaptive  e-learning 

6.  Utilize  an  upgraded  content  management  system 

7.  Increase  e-learning’s  usability  and  reliability 

8.  Update  and  enhance  course  content  review  process 

9.  Develop  meaningful  system  metrics 

1 0.  "test  and  evaluate  the  system 


the  online  database)  and  presents  easily  interpre¬ 
table,  dynamic,  and  interactive  visual  depictions  of 
the  aggregate  of  those  learning  outcomes,  as  well 
as  usable  quantitative  scores.  The  dashboard  should 
allow  OTs  and  commanders  to  easily  view  relevant 
performance  and  reaction  scores.  By  providing  an 
overview  of  aggregate  student  performance  to  the 
OTs  prior  to  the  execution  of  JELC  Academics,  for 
instance,  they  can  deliver  a  training  package  that  is 
more  tailored  to  the  training  audience’s  knowledge 
level. 

4.  Improve  the  development  and  validation  pro¬ 
cesses  for  assessment  within  CoL  courses.  ISDs, 
OTs,  and  other  training  stakeholders  recognize  the 
importance  of  creating  good  assessments;  however, 
many  times  stakeholders  are  too  busy  to  adequately 
develop  those  assessments.  Revised  processes  must 
better  support  those  stakeholders  and  content 
owners,  while  ensuring  that  courses  include  reli¬ 
able,  valid,  and  diagnostic  assessments,  designed  for 
both  formative  and  summative  evaluation  of  learn¬ 
ing  outcomes. 

5.  Implement  adaptive  e-learning.  Tailoring  in¬ 
structional  material  and  delivery  mechanisms  to 
better  meet  students’  unique  needs  enhances  learn¬ 
ing  effectiveness,  efficiency,  and  satisfaction.  In  e- 
learning,  the  software  can  automatically  adapt  the 
instruction  to  approach  each  learner  uniquely.  For 
instance,  simple  mechanisms  can  dynamically  ad¬ 


just  the  content,  learning  progression,  or  instruc¬ 
tional  activities  in  an  online  course  based  upon 
variables,  such  as  rank,  duty  assignment,  perfor¬ 
mance  on  a  course  exam,  or  the  specific  themes  of 
an  upcoming  exercise. 

6.  Utilize  an  upgraded  content  management  sys¬ 
tem.  To  support  the  adaptive  learning  mechanisms 
discussed  in  the  previous  recommendation,  the 
learning  content  management  system  must  be  up¬ 
graded  beyond  the  scope  of  many  older  DoD  con¬ 
tent  management  systems.  Similar  upgrades  are  re¬ 
quired  to  enable  the  delivery  of  more  sophisticated 
assessments  (recommendation  4),  associated  feed¬ 
back,  and  the  evaluation  and  display  of  assessment 
data  via  the  data  dashboard  (recommendation  3). 

7.  Increase  e-learning’s  usability  and  reliability. 

In  addition  to  utilizing  a  system  with  expanded  ca¬ 
pabilities,  that  system  must  also  have  high  usability 
and  reliability.  This  upgraded  content  management 
system  must  also  integrate  with  the  Service’s  learn¬ 
ing  management  systems,  as  well,  in  order  to  ensure 
that  training  outcomes  are  shared  across  platforms. 

8.  Update  and  strengthen  the  course  content  re¬ 
view  process.  A  more  robust  process  for  reviewing 
e-learning  (and  SGST  scenario)  content  and  deliv¬ 
ery  tactics  is  needed  in  order  to  ensure  all  courses 
meet  a  consistently  high  level  of  quality.  We  recom¬ 
mend  incorporating  OTs  and  other  subject-matter 
experts  (e.g.,  doctrine  specialists)  into  the  current 
review  process  so  that  courses  better  support  indi¬ 
vidual  learning — and  collective  training — goals. 

9.  Develop  meaningful  system  metrics.  As  Joint 
Training  adopts  these  recommendations  and 
evolves  into  its  “next  evolution,”  it  is  prudent  to 
develop  associated  systematic  measures  of  per¬ 
formance,  measures  of  effectiveness,  and  return- 
on-investment  criteria  to  monitor  the  immediate 
progression  of  this  effort  and  to  evaluate  long-term 
system  performance. 

10.  Test  and  evaluate  the  system.  Finally,  as  good 
system  engineering  practice  demands,  the  changes 
recommended  in  this  report  should  be  iteratively 
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designed,  developed,  tested,  and  evaluated.  As  new 
CoL  components  are  developed,  those  features 
should  be  evaluated  against  the  CoL  VI. 0  beta  test 
data.  Also,  a  periodic  (e.g.,  annual)  test  and  evalu¬ 
ation  plan  should  be  implemented  to  collect  ongo¬ 
ing  data  that  addresses  the  new  system-level  met¬ 
rics  described  in  recommendation  9. 

CoL  Next  Steps 

Based  upon  t  ^luccess  of  CoL  VI. 0, Joint  Training 
is  now  pursingmevelopment  of  CoL  V2.0,  which 
is  scheduled  to  be  completed  in  2013.  Version  2.0 
will  expand  the  content  of  the  VI. 0  CoL,  incor¬ 
porating  additional  100-  and  200-level  courses,  as 


well  as  300-level  joint  fundamentals  content.  V2.0 
will  also  include  more  sophisticated,  more  detailed 
metrics  as  well  as  visualizations  of  the  outcome 
data  designed  (via  the  “data  dashboard”)  to  give 
commanders  and  training  personnel  additional  in¬ 
sights  into  the  staff’s  individual  cognitive  readiness. 
Finally,  V2.0  will  incorporate  a  peer-learning  web- 
based  training  simulation,  called  the  Joint  Opera¬ 
tions  Center  Simulation  (JOCSIM),  using  the  ex¬ 
isting  Small  Group  Scenario  Trainer  to  provide  the 
technical  capability.  After  students  complete  their 
individual  courses  (i.e.,  the  100-300  level  courses), 
they  will  be  able  to  interact  with  fellow  person¬ 
nel  in  the  JOCSIM.  These  enhancements  will  be 
tested  during  a  2013  collective  training  event,  and 
detailed  in  a  post-event  supplementary  report. 
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GAPS  & 
OPPORTUNITIES 


This  report  describes  an  initiative,  called  the  Continuum  of  eLearn¬ 
ing  (CoL),  which  addresses  some  of  the  gaps  and  limitations  of  indi¬ 
vidual  and  collective  training  by  providing  blended  learning-train¬ 
ing  packages  to  better  integrate  online  learning  activities  into  the 
Joint  Exercise  Live  Cycle.  The  CoL  is  a  web-based  training  package, 
developed  by  Joint  Training  (Joint  Staff  J7),  and  it  is  intended  to 
represent  the  “next  evolution”  of  training  on  JKO.1 

This  section  describes  the  demand  signal  for  the  CoL.  It  lists 
five  opportunities  for  improving  collective  joint  training  and  then 
describes  the  how  individual  e-learning  can  help  achieve  those  im¬ 
provements.  However,  practical  limitations  in  military  e-learning 
often  reduce  its  effectiveness  and,  in  turn,  limit  its  ability  to  suf¬ 
ficiently  meet  collective  training  needs.  Consequently,  this  section 
ends  with  a  discussion  about  the  gaps  in  military  e-learning  and  the 
established  best  practices  for  addressing  them. 


U.S.  and  Colombian  personnel  in  the  joint 
tactical  operations  center  during  PANAMAX 
20 1 2  (Photo  by  MSG  Kevin  Doheny) 


Collective  Training 

Each  year,  the  Joint  Staff  J7  coordinates  dozens  of  large-scale  train¬ 
ing  events  for  the  Combatant  Commands  (CCMDs).  These  annual 
or  semi-annual  exercises  help  prepare  personnel  at  the  operational 
staff  level  for  their  duties  at  CCMDs  and  at  the  Joint  Task  Forces 
(JTFs)  that  support  them.  Despite  the  effectiveness  of  these  train¬ 
ing  events,  military  leaders  are  constantly  looking  for  ways  to  en¬ 
hance  their  training  outcomes.  In  particular,  joint  training  personnel 
have  identified  five  areas  that  could  be  improved  upon. 


1  Joint  Knowledge  Online  (JKO)  develops  and  delivers  online  joint  training  to  prepare 
and  assist  individuals  to  support  joint  and  coalition  operations.  JKO  is  the  DoD’s  unique, 
authoritative  capability  that  provides  operationally  relevant  and  globally  accessible  training 
to  support  individual  joint  preparedness  and  exercises.  Combatant  commands  are  depen¬ 
dent  on  JKO  as  an  integral  component  of  their  individual  joint  training  plans. 


SECTION  SUMMARY 

The  CoL  seeks  to  address 
five  gaps  related  to  collective 
training  and  staff  operations: 

(I)  staff  members  missing  out 
on  collective  training;  (2)  lack 
of  integration  among  training 
events;  (3)  lack  of  joint  mindsets; 
(4)  insufficient  individual  outcome 
data;  and  (5)  knowledge  decay 
between  training  events. 

The  CoL  is  addressing  these 
gaps  via  e-learning;  however, 
to  be  effective,  the  CoL  must 
also  address  practical  limitations 
frequently  encountered  in 
military  e-learning. 
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I.  “Untrained”  Staff 

Forming  a  joint  headquarters  staff  presents  unique 
logistical  challenges,  particularly  from  the  person¬ 
nel  and  manning  perspective.  Relevant  here  is  the 
challenge  of  ensuring  that  personnel  maximize  the 
benefit  from — and  can  actually  attend — training 
events. 

Joint  billets  are  often  filled  with  individual 
augmentees  from  across  the  Services’  reserve  com¬ 
ponents.  They  have  competing  duties,  constrained 
schedules,  and  a  “lack  of  dwell  time,”  which  “inhib¬ 
its  forces  from  conducting  the  training  necessary 
to  carry  out  the  full  range  of  military  operations 
and  to  be  prepared  for  a  wider  range  of  contingen¬ 
cies.”2  Thus,  despite  access  to  well-planned  joint 
training  events,  personnel  are  often  left  to  prepare 
just  beforehand;  they  may  struggle  during  train¬ 
ing  to  learn  the  foundational  joint  concepts,  rather 
than  use  that  time  to  hone  more  advanced  skills.  In 
other  words,  some  personnel  may  not  reap  the  full 
benefits  from  their  participation  in  joint  exercises. 

This  challenge  is  magnified  further  if  the  aug¬ 
mentees  or  late- arriving  staff  members  miss  the 
collective  training  event  altogether.  Commanders 
have  deployed  with  portions  of  their  Joint  Manning 
Document  (JMD)  still  unfilled.3  One  JTF  even  re¬ 
ported  deploying  with  less  than  50%  of  its  JMD.4 
When  this  occurs,  it  means  that  a  significant  por¬ 
tion  of  a  joint  staff  misses  the  opportunity  to  train 
with  the  unit  prior  to  deployment  and,  therefore, 
receives  less  preparation  for  their  deployment.5 

While  every  effort  is  made  to  ensure  the  qual¬ 
ity  and  authenticity  of  training,  high  percentages  of 
joint  staffs  still  do  not  receive  the  full  advantage  of 


2  Chairman  of  the  Joint  Chiefs  of  Staff  (2007).  Joint  training 
manual  for  the  armed  forces  of  the  United  States  (CJCSM  3S00.03B). 
See  p.  3. 

3  The  JMD  is  a  record  of  assigned  personnel  and  billets. 

4  Wright,  D.  P.,  &  Reese,  T.  R.  (2008).  On  Point  II:  Transition  to 
the  New  Campaign.  Combat  Studies  Institute  Press,  US  Combined 
Arms  Center.  Fort  Leavenworth,  Kansas. 

5  “It’s  not  just  that  JMD  billets  go  unfilled,”  one  of  the  PANA- 
MAX  Observer-Trainers  explained  to  the  CoL  researchers.  “Often 
someone  isn’t  qualified  for  their  assigned  billet,  and  unless  the  staff 

interviews  them  prior  to  or  just  after  arrival,  best  use  of  skills  avail¬ 
able  is  delayed  until  a  bad  operational  result  highlights  the  problem.” 


their  pre-deployment  exercises. These  personnel  re¬ 
main  “untrained”  since  no  alternative  mechanisms 
currently  substitute  for  the  collective  events.  This 
creates  weaknesses  in  the  shared  knowledge-base 
of  operational  staffs. 

2.  Stovepiped  Training 

Large-scale  joint  exercises  primarily  emphasize  in¬ 
residence  collective  training  versus  a  blend  of  indi¬ 
vidual  and  collective  approaches.  Despite  the  high 
quality  of  the  training  offered,  the  various  learning 
activities  are  currently  executed  in  stovepipes,  with 
little  correspondence  among  individual  training 
concepts,  joint  academics,  and  collective  exercise 
objectives.  To  individual  trainees,  the  flow  between 
the  stages  of  training  can  seem  disjointed,  and  each 
component  may  appear  to  lack  context. 

3.  Service-Specific  Mindsets 

Personnel  assigned  to  a  joint  billet  may  not  have 
served  in  any  joint  position  previously.  They  bring 
with  them  their  years  of  Service  experience  but  not 
necessarily  an  understanding  of  the  larger  context 
of  joint,  interorganizational,  and  multinational  op¬ 
erations.  They  may  not  know  how  to  function  in 
a  joint  manner,  and  they  may  not  yet  fully  under¬ 
stand  the  advantages  of  their  sister  Services,  the 
possible  benefits  of  Service  integration,  or  the  joint 
doctrinal  processes  for  planning  and  integration. 6 
Consequently,  personnel  may  exhibit  Service-cen¬ 
tric  attitudes  that  initially  inhibit  their  effectiveness 
in  their  new  joint  roles. 

4.  Insufficient  Constructive  Insights 

Targeted,  objective  assessment  of  personnel’s  cog¬ 
nitive  capacities  is  rarely  conducted  across  the  en¬ 
tire  cohort  of  trainees. This  causes  problems  because 
individual  augmentees  and  new  arrivals  may  carry 
with  them  unforeseen  gaps  in  critical  joint  knowl¬ 
edge;  alternatively,  they  could  possess  fife  experi¬ 
ences  that  would  distinguish  them  as  high-utility 

6  Menaker,  E.,  MacDonald,  J.,  Hendrick,  A.,  &  O’Conner,  D. 
(2006).  Training  a  Joint  and  Expeditionary  Mindset.  United  States 
Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
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officers.  Joint  training  stakeholders  must  improve 
the  way  that  they  “track  joint  training  received  by 
their  individual  personnel  so  as  to  improve  aware¬ 
ness  and  application  of  joint  trained  personnel.”7  In 
other  words,  “we  must  develop  more  effective  ways 
to  improve  and  assess  joint  and  unit  readiness.”8 

Having  enhanced  individual  readiness  data 
would  give  commanders  more  detailed,  construc¬ 
tive  insights  into  staff  members’  preparedness,  and 
observer- trainers  (OTs)  insight  into  what  aggre¬ 
gate  knowledge  gaps  can  be  covered  through  train¬ 
ing,  such  as  pre-exercise  academics.  Further,  know¬ 
ing  this  information  shortly  after  a  Service  member 
joins  a  command  would  speed  his/her  integration 
into  the  staff  and  facilitate  more  efficient  opera¬ 
tions,  overall. 

5.  Unknown  Retention  Between  Events 

Staffs  are  constantly  changing  due  to  routine  ro¬ 
tations  and  reassignments.  Consequently,  there  is 
an  ongoing  struggle  to  maintain  a  high  “band  of 
excellence”  in  the  expertise  of  the  permanent  staff 
and  to  “ensure  continuous  Joint  Mission  Essential 
Task  List  (JMETL)  refinement  to  reflect  current 
readiness,  lessons  learned,  joint  experimentation, 
capabilities  development,  training,  and  joint  mili¬ 
tary  education  programs.”9 

In  addition  to  this  unique  difficulty,  joint 
training  personnel  also  face  the  conventional  chal¬ 
lenge  of  training  transfer.  It  is  unclear  how  much 
training  transfers  to  individuals  during  a  collective 
training  event,  as  well  as  how  much  of  that  trans¬ 
ferred  knowledge  personnel  retain  between  events. 
To  compound  the  challenge,  each  staff  member  is 
likely  to  retain  distinct  parts  of  the  training  and, 
consequently,  will  require  remediation  on  different 
components  of  it. 

Online  Learning 


7  Chairman  of  the  Joint  Chiefs  of  S  2011).  2012-2015  Chair¬ 
mans  Joint  Training  Guidance  (CJCSH  ^UO.Ol),  p.  4 

8  Chairman  of  the  Joint  Chiefs  of  Staff  (2007). Joint  training  man¬ 
ual for  the  armed  forces  of  the  United  States  (CJCSM  3500.03B),  p.  4 

9  CJCS,  2012-2015  Chairmans  Joint  Training  Guidance ,  p.  2 


Table  I .  I .  Summary  of  gaps  in  military  collective  training  and  ways 
that  online  and  blended  learning  can  help  address  each  issue. 


Issue 

Resolution  Strategy 

“Untrained”  Staff 

Staff  members  frequently 
miss  the  joint  exercise  before 
a  JTF  forms  (e.g.,  due  to  late 
assignment  or  in-country 
duties) 

24/7  e-Learning  Access 

An  obvious  benefit  of 
e-learning  is  that  it  allows  24/7 
access;  so,  staff  can  receive 
some  training,  even  if  they 
miss  the  collective  exercise 

Stovepipe  Events 

Often,  learning  opportunities 
are  executed  in  stovepipes, 
without  explicit  links  to  related 
training  events 

Blended  Learning 

Use  blended  learning 
processes  to  help  ensure  that 
related  learning  opportunities 
are  integrated  together 

Service  Mindsets 

When  first  assigned  to  a 
joint  billet,  Service  members 
may  lack  joint  mindsets, 
which  initially  inhibits  their 
effectiveness  in  joint  roles 

Contextualize  Content 

Use  narrative  and  humanized 
instructional  interventions  to 
make  joint  doctrine,  culture, 
and  vocabulary,  more  relatable 
to  online  learners 

Outcome  Data 

Commanders  and  OTs  could 
make  use  of  more  detailed, 
more  diagnostic  training 
outcome  data 

Data  Dashboard 

Use  better,  more  diagnostic 
measures  to  compile 
meaningful  individual  readiness 
data  and  then  translate  this  for 
OTs  and  commanders 

Retention 

Staff  members’  skills  and 
knowledge  may  decay 
between  training  events,  and 
each  Staff  member  will  likely 
require  a  unique  remediation 
plan 

Current  and  Targeted 

Provide  enduring  online 
content,  keep  it  concurrent 
with  real-world  operations 
and  best  practices,  and  tailor 
course  assignments  to  each 
student’s  learning  needs 

Supplementary  online  courses  and  blended  learn¬ 
ing  approaches  seem  like  obvious  solutions  for  the 
challenges  outlined  above.  Using  e-learning  to  help 
bolster  collective  training  also  follows  the  Chair¬ 
mans  guidance. 10 

Online  courses  are  a  good  solution  because 


10  The  Chairman  advises  “educators,  trainers,  commanders,  learn¬ 
ing  support  elements,  information-providers,  and  information  man¬ 
agers  [to]  use  a  myriad  of  existing  techniques,  resources,  and  tech¬ 
nologies  to  facilitate  achieving  specific  learning  objectives.”  He  goes 
on  to  write,  “Joint  Knowledge  Online  (JKO)  provides  a  Joint  Indi¬ 
vidual  Training  Toolkit  of  web  enabled  individual  and  small  group 
training  products  and  services,  with  a  knowledge  management  capa¬ 
bility  that  has  real-time  reach  back  between  individual  warfighters, 
operational  staffs,  and  key  information  sources...”  See  CJCS,  Joint 
training  policy  and  guidance  for  the  armed  forces  of  the  United  States , 
p.A-7 
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Table  1 .2.  Barriers  to  effectiveness  in  military  e-learning,  summarized  from  knowledge  elicitation  with  stakeholders.  This  table  reflects 
general  issues  with  e-learning;  it  does  not  specifically  reflect  JKO  or  any  other  learning  management  system,  in  particular. 

Category 

Topic 

Learning  Content 

•  Course  content  may  lack  specificity  (i.e.,  the  content  is  “fuzzy”  and  overly  general) 

•  Courses  focus  too  heavily  on  lower-order  thinking  (e.g.,  declarative  knowledge) 

Assessments 

•  Course  assessments  lack  depth  and/or  are  poor  quality 

•  Courses  fail  to  include  useful  formative  assessments 

•  Courses  fail  to  associate  meaningful  feedback  with  assessments 

•  Courses  have  low  minimum  standards  of  performance  (i.e.,  they  do  not  require  full  mastery) 

Motivation  to  Use 

•  Insufficient  time  given  during  duty-hours  to  complete  assigned  e-learning  courses 

•  Learners  have  “just  check-the-box  as  quickly  as  possible”  attitudes  (i.e.,  task-focused  goal  orientation) 

•  Learners  perceive  answer-sharing  as  acceptable 

•  Online  courses  lack  engaging  content,  interactivity,  and/or  relevant  multimedia 

•  Students  must  repeat  known  material  frequently  (e.g.,  annual  completion  of  same  compliance  course) 

Relevance 

•  Perceived  lack  of  relevance  of  online  learning  for  actual  duties 

•  Online  courses  lack  transparent  alignment  to  doctrine  (e.g.,  UJTL) 

•  Online  courses  lack  alignment  to  future  training  or  events 

•  Online  courses  lack  concurrency  with  real-world  lessons  learned 

Usability 

•  Crashing  systems  (e.g.,  causes  lost  progress) 

•  Lack  of  interoperability  between  joint/Service  systems 

•  Slow  downloads  and  partial  downloads  prevent  completion 

they  are  available  anytime  and  anywhere,  which 
can  help  reduce  the  number  of  “untrained”  staff 
and  serve  as  ongoing  refresher  training  thereafter. 
Online  courses  can  include  elements  of  both  train¬ 
ing  and  education,  as  well  as  content  specifically 
geared  to  address  joint  mindsets.  E-learning  can  be 
readily  tailored  to  a  variety  of  training  objectives, 
in  order  to  better  prepare  personnel  for  designated 
collective  events.  Individual  performance  scores 
can  also  be  recorded  and  aggregated  to  give  com¬ 
manders  constructive  insights  into  their  personnel’s 
readiness  (see  Table  1.1  for  a  summary).  In  the  aca¬ 
demic  literature,  these  sorts  of  obvious  advantages 
are  well  documented. 

Research  also  demonstrates  that  well-de¬ 
signed  online  courses  enhance  learning  outcomes. 
For  instance,  according  to  a  recent  meta-analysis 
by  the  Department  of  Education,  in  a  review  of  50 
studies,  both  adult  and  child  learners  performed 
modestly  better  in  online  environments  as  com¬ 
pared  to  traditional  face-to-face  classroom  settings 
(Cohen’s  d  =  +0.20  in  favor  of  online  learning).  The 
same  review  found  that  courses  involving  blended 
online  and  face-to-face  instruction  performed  even 


better  (Cohen’s  d  =  +0.35  in  favor  of  blended  learn¬ 
ing  versus  face-to-face  classes).11 

Military  e-Learning  Challenges 

Only  effective  e-learning  can  significantly  enhance 
collective  training.  However,  in  many  cases,  mili¬ 
tary  online  courses  lack  engagement,  are  not  suf¬ 
ficiently  efficient,  or  seem  to  lack  relevancy.  Online 
courses  are  often  stovepiped,  without  clear  link¬ 
ages  to  other  training  and  education  events.  The 
outcome  scores  provided  by  the  courseware  do  not 
generally  give  commanders  or  military  instructors 
actionable  insights  into  personnel’s  actual  skills.  In 
other  words,  despite  the  potential  benefits  of  well- 
designed  e-learning,  online  courses  in  practice  of¬ 
ten  suffer  from  a  range  of  limitations  that  nega¬ 
tively  impact  their  effectiveness,  utility,  and  appeal. 

To  initially  identify  potential  challenges  in 
military  online  learning,  we  conducted  structured 
interviews  with  seven  active  duty  and  government 


11  U.S.  Department  of  Education,  Office  of  Planning,  Evalua¬ 
tion,  and  Policy  Development  (2010).  Evaluation  of  Evidence-Based 
Practices  in  Online  Learning:  A  Meta-Analysis  and  Review  of  Online 
Learning  Studies.  Washington,  D.C. 
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civilian  stakeholders  associated  with  the  major 
military  e-learning  enterprises  in  the  Air  Force, 
Army,  Navy,  Marine  Corps,  and  Joint  Staff.  These 
(anonymized)  subject-matter  experts  offered  in¬ 
formed  opinions  about  military  online  courses,  and 
they  described  specific  issues  that  frequently  affect 
military  online  learning.  These  are  summarized  in 
Table  1.2. 

Overall,  respondents  identified  issues  related 
to  the  depth  and  specificity  of  online  content,  as 
well  as  the  associated  assessments.  Respondents 
also  indicated  that  many  students  lack  motivation 
to  complete  online  courses  for  a  variety  of  reasons; 
similarly,  they  reported  that  some  students  believe 
that  the  courses  lack  real-world  relevance.  Finally, 
all  respondents  highlighted  various  usability  issues 
that  frustrate  and  discourage  e-learning  students. 
(It  is  important  to  note  that  this  feedback  reflects 
general  issues  with  military  e-learning;  it  does  not 
specifically  target  JKO  or  any  other  learning  man¬ 
agement  system,  in  particular.) 

e-Learning  Best  Practices 

In  order  to  shape  our  plans  for  mitigating  the  prac¬ 
tical  limitations  uncovered  in  the  interviews,  we 
compiled  relevant  best  practices  from  the  academic 
literature.  We  call  these  “practical”  limitations  be¬ 
cause  the  technology  and  academic  knowledge  to 
fix  these  problems  is  readily  available;  yet,  the  issues 
persist  mainly  due  to  logistical  limitations  (e.g., 
budgets)  or  process  problems  (e.g.,  lack  of  commu¬ 
nication  protocols). 

We  began  this  analysis  with  a  review  of  gener¬ 
al  educational  best  practices,  since  many  elements 
of  good  instruction  (e.g.,  the  importance  of  feed¬ 
back  and  utility  of  formative  assessments)  apply 
across  all  media.  Then,  building  on  the  groundwork 
of  these  general  recommendations,  we  conducted 
a  more  specific  review  of  best  practices  for  online 
learning  (also  known  as  e-learning  or  web-based 
instruction 12)  and  blended  learning  (also  known  as 
hybrid  learning).  Results  of  the  review  have  been 


12  In  this  report,  the  terms  e-learning,  online  learning,  and  •web- 
based  learning  are  used  interchangeably. 


Table  1 .3.  Summary  of  recommendations  identified  from  the 
academic  literature;  this  high-level  review  is  not  intended  to 
describe  all  e-learning  best  practices. 

Enhance  Learning  Content 

•  Clarify  expected  performance  at  the  beginning 

•  Vary  content  delivery  styles 

•  Implement  active  learning  techniques 

•  Emphasize  higher-order  thinking 

Enhance  Online  Assessments 

•  Incorporate  valid,  reliable,  and  diagnostic  assessments 

•  Use  formative  assessments  to  enhance  learning  outcomes 

•  Provide  frequent,  prompt,  and  constructive  feedback 

Enhance  Motivation 

•  Encourage  self-regulation 

•  Employ  a  mastery-learning  approach 

•  When  instructional^  appropriate,  use  multimedia 

•  Humanize  the  technology 

Enhance  Relevance 

•  Tailor  courses  to  students’  characteristics  and  learning  needs 

•  Relate  instruction  to  the  “big  picture” 

•  Use  blended  learning 

Enhance  Usability 

•  Implement  an  easy-to-navigate,  consistent  course  structure 

•  Limit  technological  glitches 

•  Provide  technical  support 


summarized  below.  For  this  discussion,  we  have 
grouped  these  recommendations  based  upon  the 
categories  identified  during  the  stakeholder  in¬ 
terviews  (although  some  of  the  recommendations 
support  several  of  these  goals).  See  Table  1.3  for  an 
outline  of  the  literature-based  recommendations. 

I .  Enhance  Learning  Content 

Clarify  expected  performance  at  the  beginning. 

Students  will  perform  better  in  any  educational 
setting  if  they  have  a  clear  understanding  of  course 
organization,  goals,  and  expectations.  In  online 
environments,  research  suggests  that  learners  are 
less  distracted13  and  achieve  higher  learning  out¬ 
comes  14  when  online  courses  have  clear  expecta- 


13  Dykman,  C.  A.,  &  Davis,  C.  K.  (2008).  Online  education  fo¬ 
rum:  Part  two — teaching  online  versus  teaching  conventionally. 
Journal  of  Information  Systems  Education,  19(2),  157-164. 

14  Zsohar,  H.,  &  Smith,  J.  A.  (2008).  Transition  from  the  class¬ 
room  to  the  web:  Successful  strategies  for  teaching  online.  Nursing 
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tions  and  guidelines.  To  facilitate  this,  introductory 
course  material  should  clearly  establish  standards 
for  performance  and  help  students  understand  the 
goals  of  the  course  and  sequence  of  learning  events. 
In  addition  to  clearly  stating  this  information  at 
the  beginning  of  a  course,  expectations  should  be 
frequently  reinforced  throughout  it,15  and  activi¬ 
ties  and  assessments  should  concretely  link  to,  and 
continuously  reinforce,  these  established  guidelines 
and  objectives. 

Vary  content  delivery  styles.  The  academic  litera¬ 
ture16  frequently  suggests  that  courses,  regardless 
of  medium,  incorporate  variety ,  such  as  various  in¬ 
structional  interventions,  in  order  to  increase  stu¬ 
dent  engagement  and  enhance  learning  outcomes. 
When  courses  fall  into  predictable,  redundant  de¬ 
livery  modes,  students  quickly  become  disengaged 
and  soon  begin  looking  for  shortcuts  so  that  they 
can  quickly  skip  from  one  predictable  task  to  the 
next.  Varying  delivery  styles  helps  hold  students’ 
attention  and  prevent  them  from  adopting  a  task- 
focused  (“just  get  it  done”)  orientation. 

Implement  active  learning  techniques.  As  noted 
by  Chickering  and  Erhmann,17  “learning  is  not  a 
spectator  sport.”  Learners  must  be  able  to  discuss, 
write  about,  and  reflect  upon  the  material  they  are 
studying.  In  other  words,  effective  curricula  encour¬ 
age  students  to  take  an  active  role  in  their  learning 
processes,  and  the  instructional  tactics  that  facili¬ 
tate  this  are  called  active  learning  techniques.18 


Education  Perspectives,  29(1),  23-28. 

15  Swan,  K.  (2003).  Learning  effectiveness:  What  the  research 
tells  us.  In  J.  Bourne  &  J.  C.  Moore  (Eds.),  Elements  of  Quality 
Online  Education,  Practice  and  Direction  (pp.  13-45).  Needham, 
MA:  Sloan  Center  for  Online  Education. 

16  For  instance,  see  the  following:  Doyle,  N.  W.  (2010).  Accom¬ 
modating  graduate  student  learning  styles  in  post-professional  online  oc¬ 
cupational  therapy  courses.  (Unpublished  doctoral  dissertation).  Bos¬ 
ton  University,  Boston,  MA.  •  Gaytan,  J.,  &  McEwen,  B.  C.  (2007). 
Effective  online  instructional  and  assessment  strategies.  American 
Journal  of  Distance  Education,  21,  117-132.  •  Rinaldi,  C.,  &  Gu- 
rung,  R.  (2008).  Should  teaching  and  learning  styles  match?  Teach¬ 
ing  Forum,  October  26, 2008  Edition. 

17  Chickering,  A.,  &  Ehrmann,  S.  C.  (1996).  Implementing  the 
seven  principles:  Technology  as  lever .AAHE  Bulletin,  49(2),  2-4. 

18  Hess,  G.  (1999).  Principle  3:  Good  practice  encourages  active 


In  contrast  to  active  learning,  passive  learn¬ 
ing  techniques  generally  involve  didactic  methods 
that  may  facilitate  factual  recall  but  fail  to  promote 
deeper  understanding.  Passive  learning  approaches 
are  unlikely  to  move  students  beyond  basic  com¬ 
prehension  to  higher-order  thought,  and  although 
learning  declarative  and  procedural  information  is 
important,  low-level  comprehension  only  partial¬ 
ly  supports  most  learning  goals.  Therefore,  active 
learning  techniques  help  learners  both  acquire  and 
extend  their  lower-level  knowledge  so  that  they  be¬ 
gin  to  understand  whole  concepts  rather  than  sim¬ 
ply  rote  definitions  and  routines. 19 

Emphasize  higher- order  thinking.  Learners 
should  be  encouraged  to  think  critically  about  a 
topic  and  to  create  a  meaningful  representation 
of  it  in  their  own  minds.  In  other  words,  students 
should  be  encouraged  to  engage  in  higher-level 
thinking,  such  as  critical  analysis  and  reflective 
judgment.20  While  active  learning  techniques  often 
encourage  such  cognitive  processes,  other  instruc¬ 
tional  design  features  also  promote  it.  For  example, 
scaffolding  principles  that  help  students  complete 
material  that  is  slightly  too  difficult  encourages  a 
“cognitive  stretch,”  which  helps  students  gradually 
build  their  reasoning  skills  as  well  as  competence 
with  the  subject  matter.21 

Similarly,  various  e-learning  delivery  mecha¬ 
nisms  can  bolster  (or  inhibit)  learners’  higher-level 
thinking.  For  example,  using  interactive  learning 
activities  and  effectively  incorporating  multime¬ 
dia  can  encourage  students  to  reach  for  a  higher 


learning.  Journal  of  Legal  Education,  49(2),  p.  401. 

19  Chickering  &  Erhmann,  Implementing  the  seven  principles: 
Technology  as  lever. 

20  Higher-order  thinking  emphasizes  those  cognitive,  affective, 
and  psychosocial  skills  that  involve  more  sophisticated  mental  pro¬ 
cesses,  such  as  analysis,  synthesis,  evaluation,  and  metacognition.  See 
Bloom,  B.  S.  (1956).  Taxonomy  of  educational  objectives:  Handbook  I: 
Cognitive  domain.  New  York:  David  McKay  Company,  Inc.;  and 
see  Krathwohl,  D.R.  (2002).  A  Revision  of  Bloom’s  Taxonomy:  An 
Overview.  Theory  into  Practice,  41(4),  212-218. 

21  Fox,  M.,  &  Helford,  R  (1999).  Advancing  the  boundaries  of 
higher  education  in  Arizona  using  the  World  Wide  Web.  Interactive 
Learning  Environments,  7(2-3),  155-174. 
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level  of  performance,22  and  simply  ensuring  that 
the  content  is  clear  and  well-organized  can  have  a 
profound  impact.23 

2.  Enhance  Online  Assessments 

Incorporate  valid,  reliable,  and  diagnostic  assess¬ 
ments.  Monitoring,  diagnosing,  and  intervening  (if 
necessary)  to  ensure  all  students  remain  on-track 
requires  the  frequent  use  of  formal  and  informal 
assessment.  Of  course,  assessments  cannot  mere¬ 
ly  be  plentiful;  they  must  also  be  effective,  which 
means  they  must  be  reliable,  valid,  and  diagnostic. 

Reliable  assessments  are  internally  consistent; 
they  function  equally  for  all  students  and  for  the 
same  student  at  different  points  in  time.  Valid  as¬ 
sessments  accurately  target  the  knowledge,  skills, 
and  attitudes  that  they  are  intended  to  measure, 
and  diagnostic  assessments  provide  specific,  mean¬ 
ingful  insights  about  students’  capabilities. 

Use  formative  assessments  to  enhance  learn¬ 
ing  outcomes.  A  common  practice  in  education 
and  training  is  to  provide  “checks  on  learning,” 
both  during  and  immediately  following  instruc¬ 
tion.  More  formally,  these  checkpoints  are  called 
formative  and  summative  assessments,  respectively. 
Formative  assessments,  which  are  administered 
during  the  learning  process,  gauge  students’  prog¬ 
ress,  indicate  whether  teaching  and  learning  activi¬ 
ties  need  to  be  modified,  and  help  improve  learner 
achievement.  These  assessments  are  typically  less 
formal  than  summative  tests  because  the  primary 
goal  of  formative  assessments  is  to  enhance  learn¬ 
ing  rather  than  to  grade  students.  In  fact,  the  actual 
scores  earned  on  formative  assessments  need  not  be 
officially  recorded,  since  performance  on  formative 
tests  is  used  to  provide  feedback  rather  than  track 
student  outcomes. 

Students  who  complete  formative  assessments 


22  Garrison,  D.  R.,  Sc  Anderson,  T.  (2003).  E-learning  in  the  21st 
century:  A  framework  for  research  and  practice.  London:  Routledge. 

23  E.g.,  see  Swan,  K.  (2001).  Virtual  interaction:  Design  factors 
affecting  student  satisfaction  and  perceived  learning  in  asynchro¬ 
nous  online  courses.  Distance  Education,  22(2),  306-331*  Collison, 
G.,  Elbaum,  B.,  Haavind,  S.,  and  Tinker,  R.  2000.  Facilitating  online 
learning:  Effective  strategies  for  moderators.  Madison,  WI:  Atwood. 


during  a  course  learn  to  recognize  and  correct  their 
errors,  and  they  build  deeper  knowledge  and  stron¬ 
ger  skills.24  The  key  to  achieving  these  benefits  is 
providing  effective  feedback.  When  used  appro¬ 
priately,  inclusion  of  formative  assessments  and 
associated  feedback  can  improve  students’  learning 
outcomes  by  20-40  percentile  points.25 

Provide  specific,  timely,  and  actionable  feedback. 

Online  learning  should  incorporate  frequent,  clear, 
and  constructive  feedback  that  is  prompt,  relevant, 
and  actionable.  Good  quality  feedback  is  also  re¬ 
flective;  it  helps  students  troubleshoot  their  perfor¬ 
mance  shortcomings  and  self-correct.26  Effective, 
continuous  feedback  correlates  highly  with  student 
satisfaction 27  and  learning  outcomes,28  and  indi¬ 
vidualized  feedback  can  enhance  students’  comfort 
levels  in  an  online  environment,  leading  to  signifi¬ 
cant  gains  in  learning  achievement.29 

When  feasible,  instructor  feedback  is  particu- 


24  Crooks,  T.J.  (1988).  The  impact  of  classroom  evaluation  prac¬ 
tices  on  students,  Review  of  Educational  Research,  58,  pp.  438-481. 

25  Ainsworth,  L.  &  Viegut,  D.  (2006).  Common  formative  as¬ 
sessments:  How  to  connect  standards-based  instruction  and  assessment. 
Thousand  Oaks,  CA:  Corwin  Press. 

26  Nicol,  D.,  &  MacFarlane-Dick,  D.  (2006).  Formative  assess¬ 
ment  and  self-regulated  learning:  A  model  and  seven  principles  of 
good  feedback  practice.  Studies  in  Higher  Education,  31(2),  199-218. 

27  Darrington,  A.  (2008).  Six  lessons  in  e-learning:  Strategies  and 
support  for  teachers  new  to  online  environments.  Teaching  English 
in  the  Two  Year  College,  35(4),  416-421.  •  Zsohar,  H.,  &  Smith,  J.  A. 
(2008) .  Transition  from  the  classroom  to  the  web:  Successful  strate¬ 
gies  for  teaching  online.  Nursing  Education  Perspectives,  29(1),  23-28. 

28  Research  consistently  demonstrates  that  consistent  and  in¬ 
formative  feedback  is  beneficial  to  learners,  for  instance,  see  Swan, 
K.  (2003).  Learning  effectiveness:  What  the  research  tells  us.  In  J. 
Bourne  &  J.  C.  Moore  (Eds.),  Elements  of  Quality  Online  Education, 
Practice  and  Direction  (pp.  13-45).  Needham,  MA:  Sloan  Center  for 
Online  Education.  •  Chickering,  A.,  &  Gamson,  Z.  (1987).  Seven 
principles  for  good  practice  in  undergraduate  education.  AAHE  Bul¬ 
letin,  40,  3-7.  •  Chickering  &  Ehrmann,  Implementing  the  seven 
principles:  Technology  as  lever.  •  Janicki,  T.,  &  Liegle,  J.  O.  (2001). 
Development  and  evaluation  of  a  framework  for  creating  web-based 
learning  modules:  A  pedagogical  and  systems  approach.  Journal  of 
Asynchronous  Learning  Networks,  5(1),  58-84. 

29  Gallien,  T.,  &  Oomen-Early,  J.  (2008).  Personalized  versus 
collective  instructor  feedback  in  the  online  classroom:  Does  type 
of  feedback  affect  student  satisfaction,  academic  performance  and 
perceived  connectedness  with  the  instructor?  International  Journal 
on  ELearning,  7(3),  463-476.  •  Black,  P.,  &  Wiliam,  D.  (1998).  As¬ 
sessment  and  classroom  learning.  Assessment  in  Education,  5(1),  7-74. 
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larly  valuable  in  an  online  course.  However,  when 
an  instructor  is  not  available  to  offer  guidance, 
learners  must  rely  on  feedback  from  the  learn¬ 
ing  management  system.  In  these  cases,  feedback 
is  generally  provided  in  response  to  assessments, 
which  reinforces  the  need  to  integrate  evaluations 
frequently  throughout  a  course.30 

3.  Enhance  Motivation 

Encourage  self-regulation.  Online  learning  works 
best  for  independent,  highly  motivated  learners 
with  time  management,  literary,  and  technological 
skills  (i.e.,  the  characteristics  of  adult  learners).31 
To  help  students  succeed  in  this  context,  curricula 
should  promote  student  self-regulation. 

Self-regulation  requires  students  to  take  con¬ 
trol  of  their  own  learning  by  monitoring  their  prog¬ 
ress  and  comparing  it  against  established  learning 
standards.32  Research  has  shown  that  it  is  possible 
to  foster  self-regulation  in  students  by  carefully 
structuring  the  learning  environment  and  encour¬ 
aging  the  use  of  metacognition,  self-monitoring, 
and  self-regulatory  practices.33  The  research  also 
notes  that  self-assessments  and  formative  assess¬ 
ments  can  promote  these  reflective  processes.  Self- 


30  Twigg,  C.  A.  (2001 ).  Innovations  in  online  learning:  Moving  be¬ 
yond  no  significant  difference.  The  Pew  Symposia  in  Learning  and  Tech¬ 
nology  2001.  Center  for  Academic  Transformation  at  Rensselaer 
Polytechnic  Institute,  Troy,  NY. 

31  Barbour,  M.  K.  &  Reeves,  T.  C.  (2009).  The  reality  of  virtual 
schools:  A  review  of  the  literature.  Computers  &  Education  52,  409- 
416. 

32  Nicol,  D.,  &  MacFarlane-Dick,  D.  (2006).  Formative  assess¬ 
ment  and  self-regulated  learning:  A  model  and  seven  principles  of 
good  feedback  practice.  Studies  in  Higher  Education,  31(2),  199-218. 
Also,  a  more  direct  definition  of  self-regulation  is  given  by  Pintrich 
and  Zuscho:  “Self- regulated  learning  is  an  active  constructive  pro¬ 
cess  whereby  learners  set  goals  for  their  learning  and  monitor,  regu¬ 
late,  and  control  their  cognition,  motivation,  and  behavior,  guided 
and  constrained  by  their  goals  and  the  contextual  features  of  the 
environment.”  See  Pintrich,  P.  R.,  &  Zuscho,  A.  (2002).  The  devel¬ 
opment  of  academic  self-regulation:  The  role  of  cognitive  and  mo¬ 
tivational  factors.  In  A.  Wigfield  &  J.  Eccles  (Eds.),  Development  of 
Achievement  Motivation  (pp.  250-271).  NY:  Academic  Press. 

33  Schunk,  D.  H.,  &  Zimmerman,  B.  J.  (Eds.).  (1994).  Self-regu¬ 
lation  of  learning  and  performance:  Issues  and  educational  applications. 
Hillsdale,  NJ:  Erlbaum.  •  Pintrich,  P.  R.  (1995). Understanding  self- 
regulated  learning.  New  Directions  for  Teaching  and  Learning,  63, 
3-12.  doi:  10.1002/tl.37219956304 


assessments  give  learners  the  opportunity  to  reflect 
on  their  performance,  which  enhances  learning  out¬ 
comes,  and  helping  learners  engage  in  realistic  self- 
assessment  has  revealed  a  marked  improvement 
in  student’s  performance  on  summative  exams.34 

Employ  a  mastery-learning  approach.  In  mastery 
learning,  performance  standards  remain  constant 
and  the  amount  of  time  different  students  require 
to  reach  mastery  is  allowed  to  vary.  This  approach 
differs  from  common  instructional  models  in 
which  all  learners  are  given  the  same  amount  of 
time  and,  often,  the  same  instructional  interven¬ 
tions,  but  their  achievement  levels  are  allowed  to 
vary.3S  In  other  words,  the  mastery-learning  ap¬ 
proach  encourages  learners  to  master  the  material, 
not  just  “pass”  the  module.36  When  given  enough 
time  and  appropriate  instruction,  90-95%  of  stu¬ 
dents  can  achieve  mastery. 37 

Online  courses  readily  support  mastery  learn¬ 
ing  because  they  allow  students  to  receive  individ¬ 
ualized  content,  experience  different  instructional 
interventions,  and  complete  modules  at  their  own 
pace.38  However,  to  encourage  students  to  adopt 
a  mastery  learning  approach — rather  than  simply 
focus  on  course  completion — e-learning  curricula 
must  actively  refocus  students  on  mastery-learning 


34  McDonald,  B.,  &  Boud,  D.  (2003).  The  impact  of  self-assess¬ 
ment  on  achievement:  The  effects  of  self-assessment  training  on  per¬ 
formance  in  external  examinations.  Assessment  in  Education,  10(2), 
209-220. 

35  Block,  J.  H.,  &  Burns,  R.  B.  (1976).  ^Aastery  learning.  Review 
of  Research  in  Education,  4,  3-49.  •  Anderson,  L.  W.  (2000).  Why 
should  reduced  class  size  lead  to  increased  student  achievement?  In 
M.  C.  Wang  &  J.  D.  Finn  (Eds.),  How  small  classes  help  teachers  do 
their  best  (pp.  3-24).  Philadelphia:  Temple  University  Center  for  Re¬ 
search  in  Human  Development. 

36  Chickering  &  Gamson,  3-7,  and  Chickering  &  Ehrmann,  2-4. 

37  Ericsson,  K.  A.  (in  press).  Adaptive  expertise  &  cognitive  readi¬ 
ness:  A  perspective  from  the  expert-performance  approach.  In  H.  F. 
O’Neil,  R.  S.  Perez,  &c  E.  L.  Baker  (Eds.),  Teaching  and  measuring 
cognitive  readiness.  Houten,  the  Netherlands:  Springer.  *  Bloom,  B.  S. 
(1968).  Learning  for  mastery.  Evaluation  Comment,  1(2),  112. 

38  Using  mastery  learning  approaches  in  e-learning  was  discussed 
in  a  literature  review;  see  Swan,  K.  (2004).  Learning  online:  Current 
research  on  issues  of  interface,  teaching  presence  and  learner  char¬ 
acteristics.  In  J.  Bourne  &  J.  C.  Moore  (Eds.),  Elements  of  Quality 
Online  Education,  Into  the  Mainstream  (pp.  63-79).  Needham,  MA: 
Sloan  Center  for  Online  Education. 
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goals  throughout  the  course. 

When  students  focus  on  mastery  learning  they 
show  higher  levels  of  engagement  and  improved 
learning  outcomes.  Alternatively,  when  students 
adopt  task-focused  attitudes  (i.e.,  a  desire  to  “just 
get  it  done”)  they  are  less  motivated  to  learn.39  In 
addition  to  enhancing  motivation,  mastery  learning 
leads  to  higher  achievement  levels,  and  at  the  end 
of  each  learning  module,  mastery  learning  ensures 
that  students  have  acquired  the  prerequisite  knowl¬ 
edge  and  skills  to  continue  through  the  course.40 

When  instructionally  appropriate,  use  multime¬ 
dia.  As  consumers  of  online  learning,  most  people 
already  realize  that  “not  all  online  courses  provide 
high  quality  learning  experiences,  as  many  consist  of 
little  more  than  books  behind  glass  with  little  or  no 
interaction.”41  In  other  words,  like  poorly  designed 
face-to-face  classes,  many  e-learning  courses  sim¬ 
ply  use  one-way,  “‘transmissive’  rather  than  ‘interac¬ 
tive’ learning  strategies.” 42  This  contributes  to  lower 
levels  of  engagement  and  can  inhibit  the  learning 
process.  Interactive  multimedia,  however,  can  help 
convey  information  in  more  engaging,  more  com¬ 
municative  ways.43  For  example,  researchers  found 
that  when  video  modes  were  used  in  lessons,  learn¬ 
ers  achieved  higher  learning  outcomes  and  had  in¬ 
creased  retention. 44 


39  The  need  to  balance  the  complementary  goals  of  course  com¬ 
pletion  and  content  mastery  was  noted  by  Davies  in  2003.  In  a 
study  of  students  enrolled  in  an  online  undergraduate  course,  Da¬ 
vies  found  that  learners  initially  intended  to  learn  the  material  in 
a  mastery  approach,  but  later  (as  due  dates  approached),  switched 
their  intent  primarily  to  course  completion.  See  Davies,  R.  S.  (2003). 
Learner  intent  and  online  courses.  The  Journal  of  Interactive  Online 
Learning,  2(1),  1-10. 

40  Anderson,  3-24. 

41  Nagel,  L.,  Sc  Kotze,T.G.  (2009).  Supersizing  e-learning:  What 
a  Col  survey  reveals  about  teaching  presence  in  a  large  online  class. 
Internet  and  Higher  Education,  p.  1 

42  Waddoups,  G.,  Sc  Howell,  S.  (2002).  Bringing  online  learning 
to  campus:  The  hybridization  of  teaching  and  learning  at  BYU.  In¬ 
ternational  Review  of  Research  in  Open  and  Distance  Learning,  2(2). 

43  see  Richard  Mayer’s  Multimedia  Theory  for  guidelines  of  how 
to  appropriately  present  different  media;  for  instance,  see  Mayer,  R. 
(2001).  Multimedia  Learning.  Cambridge:  U.  of  Cambridge. 

44  Dieker,  L.,  Lane,  H.,  Allsopp,  D.,  O’Brien,  C.,  Butler, T.,  Kyger, 
M.,  Lovin,  L.,  Sc  Fenty,  N.  (2009).  Evaluating  video  models  of 
evidence-based  instructional  practices  to  enhance  teacher  learning. 


Humanize  the  technology.  One  strength  of  tra¬ 
ditional  face-to-face  courses  is  that  they  facilitate 
important  social  interactions  with  peers  and  in¬ 
structors.  These  relationships  encourage  student 
self-assessment,  increase  their  self-efficacy45  and 
satisfaction,46  and  enhance  learning  outcomes.47  In 
online  environments,  collaboration  tools  such  as 
blogs,  wikis,  podcasts,  vodcasts,  and  virtual  worlds 
can  effectively  support  such  interactions. 

Many  e-learning  courses,  however,  are  entirely 
asynchronous,  and  learners  never  have  the  oppor¬ 
tunity  to  interact  with  instructors  or  fellow  stu¬ 
dents.48  Some  students  are  not  ready  for  this  type 
of  independent  online  learning,49  and  they  may  feel 
isolated  or  detached.50  In  these  cases,  strategies  that 
help  “humanize”  the  technology  can  help.  For  in¬ 
stance,  the  use  of  a  personalized  welcome  note  or 
video  from  an  instructor,  or  simply  the  inclusion 
of  humor  in  a  course,  can  make  students  feel  more 


Teacher  Education  and  Special  Education,  2,  180-196. 

45  For  example,  see  Castle,  S.,  Sc  McGuire,  C.  (2010).  An  analy¬ 
sis  of  student  self-assessment  of  online,  blended,  and  face-to-face 
learning  environments:  Implications  for  sustainable  education  de¬ 
livery.  International  Education  Studies,  3(3),  36-40.  •  37  Lizano-Di- 
Mare,  M.  (2009) .  Best  practices for  online  instructional  communication. 
Journal  of  Instruction  Delivery  Systems,  23(3),  17-22. 

46  For  example,  see  Appana,  S.  (2008).  A  review  of  benefits  and 
limitations  of  online  learning  in  the  context  of  the  student,  the  in¬ 
structor,  and  the  tenured  faculty.  International  Journal  on  ELearning, 
7(1),  5-22.  •  38  Gallien,T.,  Sc  Oomen-Early,  J.  (2008).  Personalized 
versus  collective  instructor  feedback  in  the  online  classroom:  Does 
type  of  feedback  affect  student  satisfaction,  academic  performance 
and  perceived  connectedness  with  the  instructor?  International Jour¬ 
nal  on  ELearning,  7(3),  463-476. 

47  For  example,  see  Means,  B., Toyama,  Y.,  Murphy,  R.,  Bakia,M., 
&  Jones,  K.  (2009).  Evaluation  of  evidence-based  practices  in  online 
learning:  A  meta-analysis  and  review  of  online  learning  studies.  Wash¬ 
ington,  DC:  U.S.  Department  of  Education.  •  Swan,  K.,  Sc  Shih, 
L.  F.  (2005).  On  the  nature  and  development  of  social  presence  in 
online  course  discussions.  Journal  of  Asynchronous  LearningNetworks, 
9(3),  115-136.  •  Gaytan,  J.,  Sc  McEwen,  B.  C.  (2007).  Effective  on¬ 
line  instructional  and  assessment  strategies.  American  Journal  of  Dis¬ 
tance  Education,  21, 117-132.  •  Stein,  Wanstreet,  Calvin,  Overtoom, 
Sc  Wheaton,  pp.  105-118.. 

48  Welsh,  Wanberg,  Brown,  Sc  Simmering,  pp.  245-258. 

49  Graham,  C.  R.,  Allen,  S.,  ScUre,  D.  (2005).  Benefits  and  chal¬ 
lenges  of  blended  learning  environments.  Encyclopedia  of  information 
science  and  technology,  1, 253-259. 

50  Rovai,  A.  P.  Sc  Wighting,  M.  J.  (2005).  Feelings  of  alienation 
and  community  among  higher  education  students  in  a  virtual  class¬ 
room.  Internet  and  Higher  Education,  8(2),  97-110. 
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connected.51 

4.  Enhance  Relevance 

Tailor  courses  to  students’  characteristics  and 
learning  needs.  In  his  classic  study,  Bloom  found 
that  students  who  received  one-on-one  tutoring 
performed  two  standard  deviations  superior  to  stu¬ 
dents  who  were  taught  via  the  conventional  group 
method.52  The  average  tutored  student  performed 
better  than  98%  of  the  control  class.  This  study,  and 
others  like  it,  demonstrate  that  human  tutors  tailor 
their  instruction  to  their  students  and  impressive 
learning  gains  often  result  from  that  interaction. 

The  best  online  learning  environments  provide 
an  individualized  experience  for  each  learner  that 
leads  to  topic  mastery.  In  instructor-led  settings, 
teachers  can  monitor  students’  individual  difference 
and  learning  progress  and  then  adjust  instructional 
interventions,  as  needed.  In  learning  environments 
without  instructors,  adaptive  software  can  substi¬ 
tute  for  some  of  these  instructor  activities. 

Adaptive  instructional  technologies  are  com¬ 
puter-based  tools  that  attempt  to  individually  tai¬ 
lor  instruction  by  diagnosing  students’  needs  and 
then  modifying  content  and  delivery  mechanisms 
to  best  support  them.  In  other  words,  computer- 
based  adaptive  instructional  systems  “attempt  to  be 
different  for  different  students  and  groups  of  stu¬ 
dents  by  taking  into  account  information  accumu¬ 
lated  in  the  individual  or  group  student  models.”53 

Online  adaptive  instructional  technology,  or 
“adaptive  hypermedia,”  usually  personalizes  course 
content  based  upon  students’  demographics  and 
test  performance.  These  adaptations  can  include 
modifications  to  the  material  or  delivery  mecha¬ 
nisms.  For  instance,  a  senior  leader  might  automat¬ 
ically  receive  a  more  in-depth  version  of  a  course’s 


51  Anderson,  D.  G.  (2011).  Taking  the  “distance”  out  of  distance 
education:  a  humorous  approach  to  online  learning.  MERLOT Jour¬ 
nal  of  Online  Learning  and  Teaching,  7(1),  http://jolt.merlot.org/ 
vol7no  1/ anderson_03 1 1  .htm 

52  Bloom,  3-16. 

53  Brusilovsky,  P.  &  Peylo,  C.  (2003)  Adaptive  and  intelligent 
Web-based  educational  systems.  International  J.  of  Artificial  Int.  in 
Ed.,  13(2-4),  159-172. 


content,  or,  based  upon  poor  assessment  results,  a 
young  lieutenant  might  be  asked  to  complete  a  re¬ 
medial  multimedia  activity. 

Relate  instruction  to  the  “big  picture.”  Frequent¬ 
ly,  students  are  skeptical  of  the  significance  of  a 
course;  however,  all  valid  instruction  has  value  be¬ 
yond  the  classroom  or  web-portal.  To  make  a  les¬ 
son’s  relevance  more  explicit,  curricula  should  in¬ 
clude  a  bridge  to  connect  learning  events  to  student 
goals  and  real-world  applications.  Many  studies 
show  that  when  students  understand  the  relevance 
of  a  course,  they  exhibit  increased  engagement54 
and  greater  motivation  to  learn.55 

Instructional  design  that  links  new  concepts 
to  students’  goals,  experiences,  and  knowledge  also 
improves  recall;  these  connections  help  students 
“...make  what  they  learn  part  of  themselves.”56 
To  achieve  this,  instructors  and/or  course  content 
should  reinforce  and  review  previously  learned  ma¬ 
terial  and  explain  how  new  concepts  relate  to  prior 
content.  Course  activities  should  also  help  students 
connect  what  they  learn  today  with  future  learning; 
for  instance,  scenarios  and  narratives  can  help  dem¬ 
onstrate  how  knowledge  might  be  applied  in  new 
situations.  Asking  students  to  self- reflect  about  the 
ways  the  course  prepares  them  for  real-life  can  also 
help  them  recognize  its  utility. 57 

Use  blended  learning.  Blended  learning  environ¬ 
ments  (also  known  as  hybrid  learning  environ¬ 
ments)  include  both  face-to-face  instruction  and 
computer-mediated  learning.58  They  combine  the 
individualized  support  and  engagement  of  regular 

54  ShernofF,  D.  J.,  Csikszentmihalyi,  M.,  Schneider,  B.,  &  Sher- 
noff,  E.  S.  (2003).  Student  engagement  in  high  school  classrooms 
from  the  perspective  of  flow  theory.  School  Psychology  Quarterly, 
18(2),  pp.  158-176. 

55  For  example,  see  Frymiera,  A.  B.  &  Shulman,  G.  M. 
(2009) .“What’s  in  it  for  me?”:  Increasing  content  relevance  to  en¬ 
hance  students’ motivation.  Communication  Education  44(1),  pp.  40- 
50. 

56  Chickering  &  Erhmann,  2-4. 

57  Teaching  as  Leadership.  (2009).  Retrieved  www.teachingas- 
leadership.org. 

58  Graham,  C.  R.,  Allen,  S.,  &Ure,  D.  (2005).  Benefits  and  chal¬ 
lenges  of  blended  learning  environments.  Encyclopedia  of  information 
science  and  technology,  1, 253-259. 
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face-to-face  meetings  with  the  flexibility  and  avail¬ 
ability  of  online  instructional  activities. 

Blended  environments  produce  better  learn¬ 
ing  outcomes  than  either  face-to-face  or  web-based 
learning  alone. This  was  documented  in  the  Depart¬ 
ment  of  Education’s  recent  meta-analysis,  which 
found  that  blended  environments  performed  sub¬ 
stantially  better  than  traditional  face-to-face  cours¬ 
es  (Cohen’s  d  =  +0.35  in  favor  of  blended  learning) 
and  modestly  better  that  solely  online  courses  (Co¬ 
hen’s  d  =  +0.05  in  favor  of  blended  learning). 59 

Blended  learning  has  many  forms.  From 
course  to  course,  the  proportion  and  timing  of  on¬ 
line  versus  face-to-face  instruction  varies,  and  the 
electronic  options  may  include  anything  from  in¬ 
dependently  accessed,  remote  e-learning  tools  to 
collaborative  virtual  environments.60  The  effective¬ 
ness  of  blended  learning  does  not  depend  upon 
a  one-size-fits-all  model;  “...creating  a  blended 
program  is  not  dependent  on  technology.  Rather 
it  is  a  process  of  problem  identification,  defining 
the  blended  model,  and  carefully  managing  and 
measuring  program  execution.”61  In  other  words, 
success  in  blended  learning  depends  upon  tailor¬ 
ing  the  available  features,  from  both  classroom  and 
website,  to  best  meet  instructional  goals  and  stu¬ 
dents’  needs. 

5.  Enhance  Usability 

Implement  an  easy-to-navigate,  consistent 
course  structure.  Research  findings  show  that  the 
organization  of  web-based  courses  substantially  af¬ 
fects  their  learning  outcomes.  If  the  students  find  a 
learning  management  system  confusing  or  difficult 
to  use,  their  learning  outcomes  and  retention  rates 
will  decline.62 

Course  content  modules  should  clearly  inform 


59  Department  of  Education,  p.  38. 

60  Rossett,  A.  8c  Sheldon,  K.  (2001) .  Beyond  the  podium:  delivering 
training  and  performance  to  a  digital  world.  San  Francisco:  Jossey- 
Bass/Pfeiffer,  p.  281. 

61  Bersin,J.  (2004).  The  blended  learning  book:  best  practices,  proven 
methodologies,  and  lessons  learned,  San  Francisco:  Wiley  8c  Sons.  p.  5 

62  Kruse,  K.  (2002),  The  benefits  and  drawbacks  of  e-learning. 
http://e-learningguru.com 


learners  of  the  content,  sequencing,  and  perfor¬ 
mance  expectations,  and  courses  should  implement 
a  consistent  and  easily  navigable  interface.63  Other 
support  features,  such  as  help  menus,  can  also  en¬ 
hance  the  perceived  usability  of  online  courses.64 
“Course  design  and  presentation  mechanisms — 
together  with  excellence  in  online  dialogue  facilita¬ 
tion — separate  the  excellent  online  course  from  the 
mediocre  or  weak  one.” 65 

Limit  technological  glitches.  Technical  difficul¬ 
ties,  such  as  error  messages  or  degraded  media,  fre¬ 
quently  disrupt  e-learning  courses.  Unsurprisingly, 
these  technological  distractions  have  been  shown 
to  increase  learner  frustration,  decrease  learner  sat¬ 
isfaction,  and  increase  attrition  rates.  The  interrup¬ 
tions  caused  by  technical  problems  can  also  make  it 
more  difficult  for  students  to  focus  on  the  instruc¬ 
tional  material,  and  these  distractions  may  nega¬ 
tively  impact  students’  cognitive  workload.  66  Thus, 
to  the  extent  possible,  it  is  important  to  limit  the 
number  of  technical  issues. 

Provide  technical  support.  Access  to  technologi¬ 
cal  support  can  limit  the  disruptiveness  of  tech¬ 
nological  glitches.  To  facilitate  improved  learning 
outcomes  and  increase  student  satisfaction,  courses 
should  make  technological  support  easily  accessi¬ 
ble.67  In  addition  to  providing  contact  information 
for  a  help  desk,  integrated  tutorials  and  organi¬ 
zational  site  maps  can  also  support  troubleshoot- 


63  “Findings  indicated  that  clarity  of  design,  interaction  with  in¬ 
structors,  and  active  discussion  among  course  participants  signifi¬ 
cantly  influenced  students’  satisfaction  and  perceived  learning,”  see 
Swan,  p.  306. 

64  Janicki,  T.,  8c  Liegle,  J.  O.  (2001).  Development  and  evalua¬ 
tion  of  a  framework  for  creating  web-based  learning  modules:  A 
pedagogical  and  systems  approach.  Journal  of  Asynchronous  Learning 
Networks,  5(1),  58-84. 

65  Collison,  Elbaum,  Haavind,  &  Tinker,  p.  1 

66  Sitzmann,  T.,  Ely,  K.,  Bell,  B.  S.  8c  Bauer,  K.  N.  (2010).The 
effects  of  technical  difficulties  on  learning  and  attrition  during  on¬ 
line  training.  Retrieved  5  Nov  2011,  from  Cornell  University,  ILR 
School  site:  http://digitalcommons.ilr.cornell.edu/articles/40 

67  Heale,  R.,  Gorham,  R.,  8c  Fournier,  J.  (2010).  An  evaluation  of 
nurse  practitioner  student  experiences  with  online  education.  Jour¬ 
nal  of  Distance  Education,  24(3),  33-46. 
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ing. 68  If  students  are  unable  to  effectively  use  course 
functions,  cannot  access  the  learning  content,  and 
feel  neglected,  they  are  more  likely  to  drop  out  of 
a  course.69 

Chapter  Summary 

This  chapter  provided  an  overview  of  issues  asso¬ 
ciated  with  joint  collective  training.  Joint  Training 
seeks  to  resolve  these  issues  by  leveraging  web- 
based  (and  blended)  learning  approaches.  How¬ 
ever,  to  do  this  effectively,  several  practical  limita¬ 
tions  associated  with  military  e-learning  must  first 
be  resolved. 

We  conducted  a  front-end  analysis,  including 


stakeholder  knowledge  elicitation  and  a  review  of 
published  best  practices,  in  order  to  identify  the 
practical  gaps  and  corresponding  resolution  strat¬ 
egies  needed  to  enhance  military  online  learning. 
From  these  efforts,  as  well  as  ongoing  testing  and 
iterative  refinement,  the  team  developed  the  Con¬ 
tinuum  of  eLearning  framework. 

The  CoL  is  designed  to  overcome  the  five  col¬ 
lective  training  challenges  described  in  this  chapter, 
while  also  enhancing  military  e-learning.  The  CoL 
represents  both  a  capability  (i.e.,  the  course  con¬ 
tent)  and  a  methodology  (i.e.,  the  implementation 
approach)  for  bolstering  joint  training  and  educa¬ 
tion.  The  specific  design  of  the  CoL  is  described  in 
more  detail  in  Chapter  2,  and  Chapter  3  describes 
a  large-scale  beta  test  of  its  effectiveness. 


68  Chou,  C.  (2003).  Interactivity  and  interactive  functions  in  web- 
based  learning  systems:  A  technical  framework  for  designers.  British 
Journal  of  Educational  Technology,  34( 3),  265-279. 

69  Rovai,  A.  P.  &  Wighting,  M.  J.  (2005).  Feelings  of  alienation 
and  community  among  higher  education  students  in  a  virtual  class¬ 
room.  Internet  and  Higher  Education,  8(2),  97-110. 
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This  section  outlines  the  envisioned  capabilities  of  the  CoL  com¬ 
ponents  as  well  as  their  corresponding  design,  development,  and 
implementation  methodology.  These  specifications  were  derived 
from  published  best  practices  (see  Chapter  1)  as  well  as  from  em¬ 
pirical  testing  and  stakeholder  knowledge  elicitation  (see  Chapter 
3).  Through  ongoing  testing,  the  ultimate  design  of  the  CoL  may 
evolve  beyond  what  this  chapter  describes;  however,  as  of  its  writing, 
this  section  provides  the  most  current  vision  for  the  CoL.  Future 
developments  will  be  described  in  supplemental  chapters. 

CoL  Components 

The  CoL  represents  both  a  methodology  for  blending  individual  and 
collective  training,  as  well  as  the  web -based  training  tools  to  support 
the  blended  learning.  Together,  these  processes  and  technologies  are 
intended  to  boost  personnel’s  knowledge  of  joint  mission-relevant 
topics  before,  during,  and  after  an  exercise  or  deployment. 

As  its  name  implies,  the  CoL  represents  a  continuum  of  train¬ 
ing  events,  which  are  intentionally  blended  together  in  order  to  mu¬ 
tually  reinforce  one  another.  The  first  stage  in  the  CoL  consists  of 
individual,  self-paced  courses  administered  by  the  JKO  learning  con¬ 
tent  management  system  (LCMS).The  second  stage  involves  small- 
group  part-task  training,  which  is  also  delivered  through  the  online 
system.  Traditional  live  training,  including  face-to-face  academics 
and  collective  exercises,  is  then  integrated  into  the  third  stage  of  the 
continuum.  See  Figure  2.1  for  a  high-level  depiction.  Each  of  these 
components  is  described  in  more  detail  below. 

Individual  Online  Courses 

The  first  component  of  the  CoL  involves  self-paced  e-learning  de¬ 
livered  through  JKO  (see  Figure  2.2).  These  online  courses  are  di- 


Sgt.  "lova  Faust,  a  CH-47  Chinook  helicopter 
mechanic,  logs  onto  a  computer  at  Camp 
Adder,  Iraq  (www.dvidshub.net) 


SECTION  SUMMARY 

The  CoL  is  both  a  methodology 
for  designing,  developing,  and 
implementing  blended  learn¬ 
ing,  as  well  as  the  technological 
capabilities  to  support  it.  The 
technological  backbone  of  the 
CoL  resides  on  Joint  Knowledge 
Online. 

The  CoL’s  continuum  of  learn¬ 
ing  includes  self-paced  online 
courses,  a  part-task  online  team 
simulation,  and  the  existing  live 
training  events,  enhanced  with 
blended  learning  approaches. 
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Figure  2. 1 .  High-level  depiction  of  the  structure  of  the  CoL  continuum,  which  spans  from  individual  learning  to  collective  training 


vided  into  three  tiers  of  instruction,  which  advance  from  level  100 
to  300.  The  100-level  lessons  are  based  on  joint  doctrine  and  contain 
joint  force  command  foundational  knowledge,  including  the  stan¬ 
dard  Joint  Force  Command  (JFC)  training  package.  The  200-level 
lessons  build  upon  the  doctrine  presented  in  the  100-level  courses. 
They  contain  experts’  operational  insights  from  published  best- prac¬ 
tices  focus  papers.  Finally,  the  300-leyeLcourses  highlight  case  stud¬ 
ies  axe.  drawn  from  the  Joint  Center  CyTperational  Analysis  -  Les¬ 
sons  Learned. 


Staff  Online  Part-Task  Training 

The  second  stage  of  the  CoL  involves  online  part-task  training  de¬ 
signed  for  use  by  small  groups  of  staff  members  (roughly  5-40  peo¬ 
ple).  This  part  of  the  CoL  simulates  staff  activities  in  a  Joint  Opera¬ 
tions  Center  (JOC)  and,  consequently,  is  called  the  JOC  Simulation 
or  JOCSIM.  The  JOCSIM  also  resides  onJKO,  and  staff  members 
use  it  to  synchronously  complete  training  scenarios,  either  in  col¬ 
located  or  distributed  settings  (see  Figure  2.3).  The  JOCSIM  is  an 
expansion  of  the  Small  Group  Scenario  Training  (SGST),  which 
CCMDs  already  use  to  facilitate  joint  training. 

Blended  Learning  and  Training 

The  third  stage  of  the  CoL  incorporates  existing  live  training,  in¬ 
cluding  large-scale  collective  exercises  as  well  as  various  supporting 
activities,  such  as  classroom-based  academics  and  small-scale  pre¬ 
training  practice  exercises  (see  Figure  2.4).  The  nature  of  these  ac¬ 
tivities  remains  largely  unchanged  in  the  CoL,  except  that  the  live 
lessons  are  blended  with  the  online  components.  Blending  allows 
the  live  training  to  more  explicitly  link  to  and  build  from  the  learn¬ 
ing  interventions  of  the  first  two  stages  of  the  CoL. 


Figure  2.2.  Screen  capture  from  an 
individual  online  course  (CoL  demo  content) 


Figure  2.3.  Screen  capture  of  an  SGST 
training  scenario  (Courtesy  of  JKO) 


Figure  2.4.  PANAMAX  20 1  I  exercise  at  US 
Army  South  (www.dvidshub.net) 
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Figure  2.5.  General  structure  of  a  CoL  individual,  self-paced  course;  the  numbers  correspond  to  the  subsections  in  the  discussion  below. 


Concept  of  Operations 

The  sections  below  describe  the  concept  of  opera¬ 
tions  (CONOPS)  for  the  CoL  online  courses  and 
JOC  simulation  scenarios,  as  well  as  the  CoL  mech¬ 
anisms  for  facilitating  blended  learning.  Although 
each  course  or  scenario  may  include  varying  in¬ 
structional  content  and  delivery  tactics,  they  should 
all  generally  adhere  to  the  high-level  structure 
and  blended  learning  techniques  discussed  below. 

Individual  Online  Learning 

The  online  courses,  which  represent  the  first  com¬ 
ponent  of  the  CoL,  are  designed  to  be  individual, 
self-paced  instructional  packages  that  students  can 
complete  without  the  aid  of  peers  or  an  instruc¬ 
tor.  Each  CoL  online  course  generally  follows  this 
structure  (see  Figure  2.5): 

1.  Opening  Vignette 

2.  Pre-Test 

3.  Content  Modules 

4.  Framing  within  Modules 

5.  Formative  Assessments  within  Modules 

6.  Use-Case  Example 

7.  Post-Test 

1.  Opening  vignette.  Each  CoL  course  begins 
with  an  engaging  multimedia  story  that  describes 
a  historic  event  related  to  the  course  content.  For 
instance,  a  module  about  joint  planning  may  begin 
by  describing  an  actual  incident  where  mission  ef¬ 
fectiveness  suffered  due  to  poor  operational  plan¬ 


ning.  These  historic  vignettes  serve  several  pur¬ 
poses.  First,  they  grab  learners’  attention,  which  is 
a  standard  principle  of  good  instructional  design. 
Similarly,  they  help  show  the  real-world  relevance 
of  courses,  and  they  better  contextualize  course 
content.  These  features,  in  turn,  enhance  students’ 
motivation,  course  relevance  ratings,  and  learning 
outcomes. 

2.  Pre-test.  Following  the  opening  vignette,  stu¬ 
dents  complete  a  pre-test.  Each  item  on  the  pre¬ 
test  corresponds  to  a  Terminal  Learning  Objective 
(TLO)  within  the  course.  To  limit  test-retest  ef¬ 
fects  and  answer  sharing,  several  versions  of  ques¬ 
tion  are  available  per  TLO,  and  the  online  JKO 
learning  management  system  randomly  selects  one 
of  these  variants  for  the  pre-test  and  the  post-test. 

An  average  CoL  course  will  have  2-10  TLOs, 
each  of  which  may  be  associated  with  1-10  En¬ 
abling  Learning  Objectives  (ELOs).  Pre-tests  in¬ 
clude  at  least  one  item  for  each  TLO;  however, 
since  most  TLOs  feature  multiple  ELOs  and  each 
ELO  aligns  with  at  least  one  question,  each  TLO 
will  likely  be  represented  by  four  or  five  items  on 
pre-  and  post-tests. 

On  the  pre-test,  a  successful  response  to  a 
given  item  suggests  that  the  student  already  pos¬ 
sess  the  knowledge  associated  with  that  TLO,  and 
therefore,  the  system  will  “opt-out”  of  those  mod¬ 
ules  in  the  course.  Scores  from  pre-tests  are  only 
used  to  inform  their  associated  online  and  blended 
courses,  and  they  do  not  adversely  affect  students’ 
permanent  records. 
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3.  Content  modules.  Student  who  fail  to  pass  all 
or  some  of  a  pre-test  must  complete  the  content 
modules  associated  with  each  failed  learning  objec¬ 
tive.  The  modules  present  their  associated  content 
using  e-learning  best  practices  that  enhance  learn¬ 
ing  experiences,  such  as  active  learning  techniques 
and  interactive  multimedia,  as  well  as  techniques 
that  enhance  learning  outcomes,  such  as  fostering 
higher-order  thinking,  self- reflection,  and  mastery¬ 
learning  attitudes. 

As  briefly  mentioned  above,  the  content  mod¬ 
ules  “align”  to  joint  force  command  publications  and 
training  requirements.  Content  from  the  100-level 
individual  online  courses,  for  instance,  is  doctrin- 
ally  focused,  and  it  explicitly  links  to  the  Universal 
Joint  Task  List  (UJTL).  Similarly,  content  in  the 
200-  and  300-level  courses  aligns  with  publications 
on  best  practices  and  operational  lessons  learned. 

4.  Framing  within  modules.  As  appropriate,  mod¬ 
ules  include  mechanisms  that  help  contextualize 
the  facts  and  procedures  being  learned.  In  other 
words,  the  modules  are  designed  to  include  content 
associations,  short  narratives,  interactive  multime¬ 
dia,  or  other  instructional  devices  that  help  frame 
the  instructional  material,  relate  it  to  real-world 
operations,  and  explain  how  it  ties  to  the  opening 
historic  vignette. 

5.  Formative  assessments  within  modules.  As  ap¬ 
propriate,  modules  include  formative  assessments 
to  evaluate  students’  progress  and  encourage  their 
self-reflection.  The  formative  assessments  are  spe¬ 
cifically  designed  to  target  higher-order  thinking, 
such  as  appreciating  how  different  concepts  fit  to¬ 
gether. 

Similar  to  the  pre-tests,  the  actual  scores  asso¬ 
ciated  with  formative  assessments  help  inform  the 
learning  process,  and  they  do  not  adversely  affect 
students’  permanent  records.  However,  these  data 
provide  opportunities  for  the  learning  manage¬ 
ment  system  to  provide  feedback.  As  feasible,  the 
system  should  provide  detailed  responses  to  each 
students’  unique  performance. 

6.  Use-case  example.  After  students  complete  all 
of  the  required  modules,  they  will  review  a  real 


or  fictional  scenario  that  shows  how  the  material 
presented  in  the  course  could  be  applied  in  a  prac¬ 
tical  use-case.  Where  the  historic  vignette  at  the 
beginning  of  a  course  is  intended  to  gain  students’ 
attention  and  increase  their  motivation  to  learn, 
the  use-cases  at  the  end  of  a  course  are  intended 
to  facilitate  students’  transfer  of  training  by  help¬ 
ing  them  learn  how  to  transform  their  new  knowl¬ 
edge  into  operational  practice.  When  appropriate, 
the  narrative  elements  within  the  use-case  example 
are  intended  to  link  to  the  associated  small-group 
training  scenario  (described  in  the  next  section  of 
the  report).1 

7.  Post-test.  Finally,  students  complete  a  post-test, 
which  includes  the  same  general  content  as  the  pre¬ 
test  (although  the  specific  questions  may  differ).  As 
part  of  a  mastery-learning  approach,  students  are 
required  to  achieve  high  levels  of  performance  on 
the  post-test;  they  must  score  at  or  near  100%  cor¬ 
rect.  If  students  miss  a  question,  they  receive  feed¬ 
back  and  have  the  opportunity  to  review  the  associ¬ 
ated  module(s).  Students  must  re-take  the  portions 
of  the  post- test  that  they  do  not  initially  pass  (with 
different  questions  per  topic),  and  students  receive 
certification  of  course  completion  only  after  com¬ 
pletely  passing  this  summative  examination. 

Staff  Online  Part-Task  Training 

The  small-group  online  simulation  component  of 
the  CoL,  or  JOCSIM,  facilities  part-task  team 
training.  Scenarios  are  designed  to  be  completed 
by  5-40  trainees  while  being  monitored  by  several 
OTs.  Because  the  software  resides  online,  the  train¬ 
ees  and  OTs  may  or  may  not  be  located  in  the  same 
physical  place. 

The  JOCSIM  is  an  expansion  of  the  Small 


1  Although  each  CoL  course  includes  only  a  single  use-case  sce¬ 
nario,  multiple  use-cases  would  ideally  be  developed  for  each  cur¬ 
riculum.  Then  different  use-cases  can  be  made  available  to  different 
learners,  based  upon  individual  students’  characteristics  (e.g.,  rank 
or  functional  area)  or  a  group’s  exercise  learning  objectives.  For  ex¬ 
ample,  a  course  on  joint  planning  may  be  associated  with  three  use- 
case  scenarios:  one  involving  migrant  operations,  another  involving 
disaster  relief,  and  a  third  associated  with  counter-drug  operations. 
Although  each  use-case  would  offer  a  walkthrough  of  joint  opera¬ 
tional  planning,  they  would  each  do  so  in  a  different  context. 
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Figure  2.6.  General  structure  of  a  CoLJOCSIM  small-group  scenario;  the  numbers  correspond  to  the  subsections  below. 


Group  Scenario  Trainer  (SGST),  an  established 
desktop  trainer  that  simulates  operational  plan¬ 
ning  activities.  The  existing  SGST  1.0  capability  is 
more  limited  than  the  ultimate  vision  for  the  JOC- 
SIM;  however,  even  in  its  current  state,  the  SGST 
successfully  supports  staff  training.2  Building  the 
SGST  into  the  complete  JOCSIM  capability  is  ex¬ 
pected  to  further  enhance  these  benefits. 

Each  JOCSIM  learning  experience  should 
generally  follow  this  structure  (see  Figure  2.6): 

1.  Opening  Vignette 

2.  Narrative  Scenario  Introduction 

3.  Training  Scenario 

4.  Post-Test 

Note:  Figure  2.6  does  not  explicitly  depict  an  after¬ 
action  review  component  because  the  OTs  are  ex¬ 
pected  to  deliver  formative  feedback  and  facilitate 
summative  AARs,  and  their  feedback  delivery  pro¬ 
cesses  are  not  constrained  by  the  formalized  CoF 
design  framework. 

1.  Opening  vignette.  Similar  to  the  individual 
online  courses,  each  JOCSIM  episode  executed 
through  the  SGST  begins  with  a  multimedia  pre¬ 
sentation  that  reinforces  the  lessons  learned  in  the 
corresponding  self-paced  course.  This  serves  to 
gain  students’  attention,  reinforce  the  previously 
learned  material,  and  prime  students’  performance 
for  the  upcoming  training  scenario.  These  opening 
vignettes  do  not  include  narrative-specific  infor¬ 
mation;  instead,  they  emphasize  general  principles 
related  to  the  training  objectives.  This  allows  reuse; 
the  same  opening  vignette  can  support  different 
training  scenarios,  so  long  as  each  scenario  includes 


2  The  Lead  Observer-Trainer  for  Unified  Endeavour  recently  re¬ 
marked  that  the  “SGST  seems  to  be  the  gap  filler  we  have  been 
looking  for  between  academics,  which  is  the  crawl,  to  MRX  [Mis¬ 
sion  Rehearsal  Exercise],  which  is  the  run.” 


the  same  high-level  training  goals  (e.  <g.,  joint  plan¬ 
ning  for  disaster  relief  and  joint  planning  for  coun¬ 
ter-drug  operations). 

2.  Narrative  scenario  introduction.  After  the 
opening  vignette  reinforces  the  training  content,  a 
second  multimedia  vignette  presents  narrative  in¬ 
formation  that  frames  the  training  scenario.  In  this 
vignette,  a  “commander”  presents  the  content  in  a 
format  similar  to  a  mission  briefing.  When  feasible, 
this  commander  may  be  a  digital  version  of  one  of 
the  trainees’  actual  senior  leaders;  in  other  cases, 
the  multimedia  commander  may  be  a  fictional 
character.  In  either  case,  the  commander  presents 
the  “road  to  crisis”  (i.e.,  the  events  that  preceded 
the  current  scenario),  as  well  as  his/her  expected 
mission  outcomes  and  general  guidance  (i.e.,  the 
instructional  goals  and  “rules”  for  the  scenario). 

3.  Training  scenario.  Once  students  have  watched 
the  introductions,  they  can  begin  the  training  sce¬ 
nario.  During  the  scenario,  participants  interact 
with  desktop  computers  similar  to  the  ones  on 
which  they  would  perform  their  normal  staff  op¬ 
erations.  Each  participant  has  access  to  realistic 
planning  interfaces  and  files,  and  simulated  injects 
(e.g.,  messages  or  news  stories)  introduce  new  in¬ 
formation  into  the  scenario  as  it  unfolds. 

4.  Post-test.  Finally,  students  individually  com¬ 
plete  a  post-scenario  assessment,  which  includes 
questions  similar  to  those  found  on  the  summative 
online  exam.  This  post-test  not  only  reinforces  the 
training  objectives  from  the  self-paced  courses  and 
small-group  scenario,  but  it  also  helps  OTs  track 
personnel’s  learning  progression  over  time. 
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Blended  Learning  and  Training  Processes 

The  third  stage  in  the  CoL  learning  continuum 
involves  live  training.  Although  the  CoL  does  not 
directly  modify  these  activities,  it  provides  mecha¬ 
nisms  for  blending  the  individual  online  courses 
and  small-group  simulation-based  training  with 
the  live,  collective  training  activities.  As  of  the 
writing  of  this  report,  the  CoL’s  blended  learning 
approach  is  being  actively  designed;  consequently, 
the  specifications  below  will  be  expanded  upon  in 
future  publications.  To  date,  however,  two  processes 
have  been  identified  to  support  CoL  blended  learn¬ 
ing  and  training  goals: 

1 .  Blended  learning-training  packages 

2.  Access  to  analyzed  outcome  data 

1.  Blended  learning-training  packages.  Blended 
learning-training  packages  are  curricula  focused 
around  a  particular  training  objective  that  include 
linked  activities  for  online  learning,  small-group 
scenarios,  and  collective  training  activities.  A  criti¬ 
cal  aspect  of  these  packages  is  that  the  components 
“align”  with  each  other  (as  well  as  with  foundation¬ 
al  knowledge,  such  as  doctrine  or  published  best 
practices).  In  practice,  this  means  that  the  CoL 
components  within  a  blended  package  must  incor¬ 
porate  the  training  themes  associated  with  a  col¬ 
lective  exercise.  For  instance,  an  effectively  aligned 
JOCSIM  scenario  would  build  from  the  training 
objectives  specified  in  a  related  online  CoL  course 
while  also  integrating  tasks  from  the  Mission  Es¬ 
sential  Task  List  (METL)  from  the  associated  col¬ 
lective  training  exercise. 

2.  Access  to  learning  outcomes  data.  As  part  of 
the  blending  process,  OTs  and  commanders  need 
access  to  personnel’s  performance  scores  from  the 
online  components  of  the  CoL.  This  allows  them 
to  better  tailor  training  activities  to  personnel’s 
knowledge  and  skill  needs. 

Enhanced  Development  Processes 

The  final  two  components  of  the  CoL  have  no 
technological  or  material  components;  they  solely 
involve  updated  processes.  The  first  set  of  updated 


processes  involves  enhancements  to  instructional 
design  and  development,  including: 

1.  Revised  content  design  processes 

2.  Revised  training  objective  ID  processes 

3.  Improved  course  alignment  processes 

4.  Processes  for  an  OT  feedback  loop 

As  with  the  blended  learning  and  training  dis¬ 
cussed  above,  the  nature  of  the  enhanced  devel¬ 
opment  processes  is  being  actively  refined.  Future 
publications  will  expand  upon  these  specifications. 

Enhanced  Execution  Processes 

The  last  CoL  component  involves  strategies  for 
enhancing  the  efficiency  of  CoL  blended  learn¬ 
ing  and  training  execution.  Again,  these  processes 
are  currently  under  development,  but  they  include 
items  such  as  streamlining  the  student  log-in  pro¬ 
cesses  for  accessing  exercise-specific  online  training 
and  providing  information  to  commanders  about 
course  completion  times  along  with  recommenda¬ 
tions  for  possible  on-duty  training  time  allotments. 

CoL  Version  1.0 

Version  1.0  of  the  CoL  was  implemented  in  2012 
through  JKO.  To  date,  it  includes  individual  online 
courses  with  content  from  the  100  (i.e.,  “basic”)  and 
200  (i.e.,  “intermediate”)  levels  of  the  joint  force 
command  curriculum,  which  covers  topics  such  as 
joint  fundamentals,  joint  planning,  and  interorga- 
nizational  and  multinational  coordination.  Dur¬ 
ing  2012,  CoL  developers  focused  on  building  and 
testing  the  enhanced  self-paced  courses,  as  well  as 
designing  the  conceptual  framework  and  processes 
for  the  overall  learning  continuum.  Although  it 
does  not  yet  achieve  the  full  vision  for  the  CoL, 
Version  1.0  incorporates  many  of  the  best  practices 
describe  in  Chapter  1  in  order  to  better  support 
student  learning. 

Enhanced  Learning  Content 

The  CoL  VI. 0  incorporates  some  of  the  content 
and  delivery  best  practices  described  in  Chapter  1, 
including  the  use  of  scaffolding,  multimedia,  and 
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(limited)  situated  learning.  Future  versions  of  the 
CoL  self-paced  courses  will  expand  upon  these 
features  to  better  clarify  learning  expectations  from 
the  start,  include  a  greater  variety  of  delivery  styles, 
and  offer  more  opportunities  for  learners  to  engage 
in  higher-level  cognitive  processes.  Additionally, 
as  the  JOCSIM  component  is  integrated  into  the 
CoL,  students  will  be  able  to  participate  in  active 
learning. 

Enhanced  Online  Assessments 

Formative  assessments.  The  CoL  VI. 0  incorpo¬ 
rates  formative  assessments  into  about  25%  of  its 
modules  (inclusion  of  more  formative  assessments 
was  deemed  too  time-consuming  for  students). 
These  quizzes  are  designed  to  enhance  learning  by 
encouraging  students  to  reflect  on  their  knowledge 
and  think  about  the  feedback  given  after  the  quiz¬ 
zes.  The  actual  scores  associated  with  the  formative 
assessments  are  not  saved  in  the  learning  manage¬ 
ment  system. 

Valid  and  diagnostic  assessments.  One  of  the  ob¬ 
jectives  of  the  CoL  is  to  engender  higher-order 
thinking.  To  motivate  students  to  engage  in  deeper 
thinking  during  CoL  courses,  and  in  order  to  as¬ 
sess  whether  they  are  meeting  learning  goals,  cor¬ 
responding  higher-order  measurement  approaches 
must  be  employed  for  both  the  formative  and  sum- 
mative  assessments. 

Many  common  apparatus,  however,  only  mea¬ 
sure  lower-level  skills.  For  instance,  tests  may  sim¬ 
ply  measure  recognition  (e.g.,  select  the  right  vo¬ 
cabulary  word  from  a  short  list  of  multiple  choice 
options),  recall  (e.g.,  given  a  short  definition,  de¬ 
termine  whether  it  is  true  or  false),  or  basic  proce¬ 
dural  application  (e.g.,  correctly  number  the  order 
of  steps  associated  with  a  given  task). 

Researchers,  however,  have  developed  a  va¬ 
riety  of  approaches  for  better  assessing  higher- 
order  skills;  these  include  the  use  of  Behaviorally 
Anchored  Rating  Scales  (BARS),  rubrics,  con¬ 
cept  maps,  card  sorting  tests,  Situational  Judg¬ 
ment  Tests  (SJTs),  metacognitive  prompts,  and 
self/team-correction.  CoL  VI. 0  employs  several 


of  these  assessments  in  creative  ways  that  function 
via  the  instructor-less  e-learning.  These  include  the 
following: 

•  Concept  maps  with  drop-down  boxes 

•  SJTs  designed  as  multiple-choice  tests 

•  Card  sorting  using  radio  buttons 

•  Open-ended  (i.e.,  text  areas)  metacognitive 
prompts  that  are  not  graded,  but  instead  fa¬ 
cilitate  formative  (self-)assessment 

Future  versions  of  the  CoL  will  include  addi¬ 
tional  higher-order  assessment  approaches,  as  well 
as  more  dynamic  feedback  mechanisms. 

Enhanced  Motivation 

Successful  curricula  engage  students  and  present 
material  in  a  way  that  helps  learners  contextualize 
the  content.  To  better  frame  the  online  learning 
material,  each  100-level  CoL  VI. 0  course  begins 
with  a  multimedia  vignette  about  a  historic  (or  his¬ 
torically  based)  event  that  established  the  need  for 
the  joint  policies  that  the  rest  of  the  lesson  covers. 
These  historical  vignettes  are  intended  to  convey 
the  relevance  of  the  course  material,  describe  the 
rationale  behind  the  joint  doctrine,  and  engage  stu¬ 
dents  through  the  use  of  interactive  media,  story¬ 
telling,  and  history. 

Mastery  learning.  In  the  CoL  VI. 0,  summative 
assessments3  are  conducted  immediately  follow¬ 
ing  each  course.  The  test  items  for  each  summative 
assessment  are  associated  with  course  TLOs  and 
ELOs,  and  post-test  items  are  randomly  select¬ 
ed  from  the  same  test  bank  as  the  pre-test  items. 
Students  who  did  not  already  pass  the  course  at 
the  pre-test  stage  must  successfully  complete  the 
summative  assessment  in  order  to  pass  each  CoL 
course.  The  CoL  employs  a  mastery  learning  ap¬ 
proach.  As  such,  in  VI. 0  of  the  CoL  students  were 
required  to  score  100%  on  post-tests  in  order  to 
successfully  complete  a  course. 


3  Summative  assessments  are  formal  tests  used  to  measure  cu¬ 
mulative  learning  outcomes,  such  as  at  the  end  of  a  course.  They 
facilitate  the  go/no-go  decision  on  whether  a  student  has  adequately 
completed  the  course,  and  they  reinforce  the  knowledge  and  skills 
gained  throughout  a  curriculum. 
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Enhanced  Relevance 

In  the  first  version  of  the  CoL,  some  personaliza¬ 
tion  was  incorporated.  Students  begin  courses  by 
completing  a  diagnostic  knowledge  test  and,  de¬ 
pending  upon  their  scores,  are  able  to  take  or  skip 
certain  modules.  This  diagnostic  not  only  tailors 
course  content  to  each  trainee’s  knowledge  and 
information  gaps,  but  to  a  limited  extent,  it  also 
enhances  students’  motivation  by  allowing  them  to 
omit  training  they  have  already  mastered.  Future 
versions  of  the  CoL  will  feature  additional  adaptive 
learning  components. 

Enhanced  Usability 

To  enhance  usability,  students  received  a  system¬ 
generated  email  that  included  a  secure  link  to  the 


online  courses.  For  the  CoL  VI. 0,  each  partici¬ 
pant’s  name  was  manually  entered  into  the  system, 
which  then  generated  the  introductory  email. 

Chapter  Summary 

This  chapter  described  the  conceptual  design  and 
concept  of  operations  for  the  CoL,  which  includes 
web-based  instructional  technologies,  blended 
learning  mechanisms,  and  associated  development 
and  execution  processes.  This  chapter  ended  with 
a  brief  description  of  Version  1.0  of  the  system, 
which  primarily  included  prototypes  for  the  in¬ 
dividual  online  courses.  The  effectiveness  of  these 
prototype  components  was  evaluated  in  a  beta  test 
at  PANAMAX  2012,  and  this  is  described  in  the 
next  chapter. 
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CoL  VI. 0 
BETA  TEST 


PANAMAX  2012  (PMX12),  a  US  Southern  Command  (US- 
SOUTHCOM)  multinational  training  exercise,  provided  an  op¬ 
portunity  to  beta  test  the  first  version  of  the  CoL.  Data  collection 
took  place  20  May-17  Aug  2012,  and  throughout  the  beta  test,  the 
research  team  documented  the  efficacy  of  the  CoL  through  a  multi¬ 
part  experiment,  examining  the  learning  effectiveness  of  the  courses 
as  well  as  their  usability,  motivational  effects,  operational  relevance, 
and  ability  to  engender  a  “joint  mindset.”The  team  also  documented 
the  extent  to  which  courses  impacted  the  staff  members’  PMX12 
operational  performance. 

Method 

PANAMAX  Exercise 


U.S.  Army  Lt.  Col  Ricardo  Bautista  briefs 
military  officials  at  the  start  of  Panamax  20 1 2 
(Photo  by  Sgt.  I  st  Class  Alex  Licea) 


PANAMAX  is  an  annual  training  event  in  which  over  a  dozen 
countries  participate.  The  exercise  involves  the  Panama  Canal  and 
typically  features  scenario  elements  involving  illegal  trafficking,  drug 
trafficking,  terrorism,  and  natural  disasters.  Personnel  from  nearly  20 
countries  participated  in  this  year’s  PANAMAX.  These  interorga- 
nizational  and  multinational  staff  members  collectively  addressed  a 
variety  of  simulated  threats  and  practiced  their  planning  and  coordi¬ 
nation  skills  during  the  week-long  exercise  in  August  2012. 

Research  Design 

The  beta  test  compared  learning  outcomes  and  feedback  obtained 
from  US  PMX12  participants  who  completed  the  CoL  online  beta 
modules  (i.e.,  the  experimental  group)  to  those  participants  who  did 
not  (i.e.,  the  control  group).  This  evaluation  examined  the  courses’ 
learning  effectiveness,  ability  to  engender  deeper  thinking,  learning 
efficiency,  operational  relevance,  motivational  qualities,  and  usability. 


SECTION  SUMMARY 

CoL  VI  .0  was  empirically 
evaluated  at  PANAMAX  20 1 2 
(N  =  196  US  personnel). 

Participants  rated  their  previ¬ 
ous  online  learning  experiences 
slightly  below  “neutral”  on  a  five- 
point  scale  (M  =  2.87). 

Participants  rated  the  CoL  slight¬ 
ly  above  “neutral”  on  a  similar 
five-point  scale  (M  =  3.24). 

Participants  who  completed  the 
CoL  scored,  on  average,  2 1  % 
better  on  the  knowledge  test 
versus  participants  in  the  control 
group  (p  =  .001). 
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Post-test  Equivalent  Groups 

Experimental  R*  O,  XCoL  02  Oa  XpMX  04 

Control  R  03  XpMX  04 


R  =  Randomization  O  =  Measure  X  =  Intervention 


Figure  4. 1 .  Experimental  diagram  of  the  CoL  study.  The  R*  signifies 
that  nonrandom  factors  may  have  unintentionally  influenced  initial 
selection  of  eligible  experimental  participants.  Military  leadership 
necessarily  determined  which  participants  were  able  to  participate 
in  the  experimental  group.  This  may  have  created  a  nonrandom 
population;  further,  members  of  this  population  could  self-select  out 
of  the  experiment  by  attrition  (i.e.,  not  completing  the  online  beta 
courses).  However,  once  integrated  into  the  experiment,  individual 
participants  were  targeted  randomly.  0:  and  02  represent  online 
testing  opportunities,  while  Xco|  signifies  the  online  beta  courses. 
03  represents  those  measurement  opportunities  that  occurred 
just  prior  to  the  PANAMAX  exercise,  and  04  indicates  those  that 
occurred  immediately  following  it,  while  XpMX  represents  the 
exercise,  itself. 

Additionally,  within  the  experimental  and  control 
groups,  we  also  assessed  potential  demographic  dif¬ 
ferences  related  to  rank  and  service. 

The  study  used  a  experimental  post-test  de¬ 
sign,  with  qualitative  surveys  and  objective  tests 
given  to  equivalent  experimental  and  control 
groups.  As  shown  in  Table  4.1,  data  were  collected 
at  multiple  dates,  however,  not  all  of  these  dates 


reflect  repeated  measures.  Instead,  some  simply 
represent  different  scheduled  opportunities  to  in¬ 
teract  with  various  portions  of  the  population.  To 
clarify,  Figure  4.1  shows  a  simplified  experimental 
diagram,  and  Figure  4.2  visually  depicts  how  the 
CoL  components  interacted  with  the  PANAMAX 
2012  Joint  Exercise  Life  Cycle  (JELC). 

Measures 

Participants  completed  a  variety  of  knowledge  and 
reactions  questionnaires  at  several  points  before 
and  after  the  PANAMAX  exercise.  The  experi¬ 
mental  design  is  illustrated  in  Table  4.1.  This  sub¬ 
section  describes  each  of  these  apparatus  in  detail. 

Demographics.  A  demographics  survey  asked  a 
range  of  biographical  questions,  including  experi¬ 
ence  with  joint  exercises,  interagency/ multinational 
planning,  and  related  training.  All  participants  re¬ 
ceived  the  same  demographics  questionnaire,  either 
at  time  Oj  for  the  experimental  group  or  time  03  for 
the  control  group  (see  Figure  4.1).  We  hypothesized 
that  some  demographic  variables  might  contribute 
to  success  on  the  knowledge  tests  or  in  framing  per¬ 
sonal  reactions;  hence,  these  data  were  analyzed  as 
potential  covariates  across  the  other  questionnaires. 


Table  4. 1 .  Experimental  design  overview  showing  data  collection  opportunities  (top),  participant  groups  (left),  and  the  data  collection 
interventions  designed  to  target  each  combination 
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Figure  4.2.  Simplified  version  of  the  PANAMAX  20 1 2  Joint  Exercise  Life  Cycle  (JELC),  with  CoL  beta  test  events  inserted.  The  white 
boxes  on  the  timeline  depict  the  standard  exercise  events,  from  planning  (on  the  left)  through  to  exercise  execution  (on  the  right).  The 
boxes  above  the  timeline  show  CoL  activities,  including  pre-exercise  planning,  delivery  of  the  online  courses,  and  data  collection. 


Content  knowledge  tests.  Experimental  partici¬ 
pants  completed  pre-  and  post-course  knowledge 
tests  prior  to  and  immediately  following  the  CoL 
courses  (i.e.,  at  times  Oj  and  O,  on  Figure  4.1). 
These  knowledge  tests  are  part  of  each  CoL  course, 
regardless  of  this  study,  but  we  also  used  their  out¬ 
come  data  to  inform  our  analysis. 

In  addition,  all  participants  completed  a  cu¬ 
mulative  knowledge  test  prior  to  the  exercise  (at 
time  O ,) .  The  control  and  experimental  group  tests 
were  identical  and  covered  a  range  of  interagency/ 
multinational  planning  topics.  These  18 -item  mul¬ 
tiple  choice  tests  were  intentionally  designed  to  be 
very  challenging  in  order  to  avoid  ceiling  effects. 
All  the  tests  were  scored  and  reported  as  percent 
correct  out  of  100%. 

Opinions  towards  previous  e-learning  survey. 

All  participants  completed  the  same  survey  regard¬ 
ing  their  past  experiences  with  military  e-learning, 
at  time  Oj  for  the  experimental  group  and  time  03 
for  the  control  group.  The  questionnaire  asked  par¬ 
ticipants  to  rate  19  items  on  a  5-point  scale  ranging 
from  “never”  to  “always.”  Items  addressed  concepts 
related  to  effectiveness  (e.g.  “online  courses  made 
me  think  deeply,  like  engaging  in  creative  problem¬ 
solving”),  relevance  (e.g.,  “online  courses  were  in¬ 
tentionally  linked  to  specific  operational  events  or 
future  missions”),  efficiency  (e.g.,  “online  courses  re¬ 
quired  me  to  re-take  material  that  I  already  knew”), 
and  satisfaction  (e.g.,  “online  course  content  held 


my  interest”). 

Participants  also  responded  to  five  open-ended 
questions,  which  asked  them  to  identify  gaps  in 
military  online  learning,  describe  how  effectively 
military  online  courses  encourage  the  development 
of  critical  thinking  skills,  offer  opinions  of  pre-tests, 
assess  the  usefulness  of  participating  in  online  sim¬ 
ulation,  and  provide  general  recommendations  for 
improving  online  courses.  Individual  responses  re¬ 
lated  to  each  concept  were  aggregated  and  reported 
on  a  5 -point  scale,  and  the  open-ended  questions 
were  assessed  for  qualitative  trends. 

Post- CoL  reactions  survey.  Experimental  partici¬ 
pants  completed  a  reactions  questionnaire  about 
the  CoL  upon  finishing  the  CoL  beta  courses,  i.e., 
at  time  0,.  The  questionnaire  asked  each  partici¬ 
pant  to  rate  21  items  on  a  5 -point  scale,  ranging 
from  “strongly  disagree”  to  “strongly  agree.”  These 
items  correspond  with  the  opinions-towards-on- 
line-learning  survey,  described  above.  The  items 
addressed  concepts  related  to  effectiveness  (e.g.,  “the 
online  course  prepared  me  for  operating  in  a  joint/ 
multinational  environment”),  relevance  (e.g.,  “the 
online  course  was  relevant  to  my  operational  du¬ 
ties”),  efficiency  (e.g.,  “the  training  course  required 
too  much  time  to  complete”),  and  satisfaction  (e.g. 
“the  online  course  content  held  my  interest”). 

The  questionnaire  also  asked  participants  to 
respond  to  three  open-ended  questions  about  the 
most  and  least  important  topics  covered  in  the  on- 
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line  courses,  any  technical  difficulties  experienced, 
and  any  additional  comments  relevant  to  the  on¬ 
line  courses.  Individual  responses  related  to  each 
concept  were  aggregated  and  reported  on  a  5 -point 
scale,  and  the  open-ended  questions  were  assessed 
for  qualitative  trends. 

Joint  mindset  survey.  All  participants  completed  a 
survey  regarding  their  experiences  with  and  opin¬ 
ions  on  working  in  joint  environments  at  time  Or 
The  questionnaire  asked  participants  to  rate  19 
items  on  a  5 -point  scale,  ranging  from  “weekly”  to 
“never.”  Items  addressed  knowledge  of  joint  princi¬ 
ples  (e.g.,  “Learn  new  joint  principles  and  proce¬ 
dures”),  knowledge  of  joint  procedures  (e.g.,  “Outline 
specific  ways  that  an  OPLAN  or  CONPLAN  af¬ 
fects  others  outside  the  US”),  education  on  core  joint 
ideas  (e.g.,  “Receive  formal  instruction  addressing 
joint  principles  and  procedures  in  class  or  online”), 
interpretation  of  joint  concepts  (e.g.,  “Do  you  have  a 
different  interpretation  than  coworkers  of  joint  val¬ 
ues  and  ethics”),  and  service-mindedness  (e.g.,  “Use 
own-Service  specific  language  or  slang  around  co¬ 
workers  from  other  services”). 

The  questionnaire  also  included  five  open-end¬ 
ed  questions  about  the  definition  of  a  “joint  mind¬ 
set,”  traits  important  to  a  “joint  mindset,”  comfort 
in  using  joint  terminology,  preparedness  to  com¬ 
municate  with  interagency  and  multinational  indi¬ 
viduals,  and  comfort  with  joint  policies  and  proce¬ 
dures.  All  of  the  scaled  questions  were  aggregated 
and  the  overall  average  was  reported  on  a  5 -point 
scale.  The  open-ended  questions  were  assessed  for 
qualitative  trends. 

Post-academics  reactions  survey.  All  participants 
received  similar,  but  slightly  different,  reaction 
questionnaires  at  time  03,  in  order  to  collect  feed¬ 
back  about  their  experience  with  the  traditional 
face-to-face  PMX12  academic  sessions. 

The  experimental  participants’  version  of  the 
survey  asked  them  to  rate  12  items  on  a  5-point 
scale,  ranging  from  “strongly  disagree”  to  “strongly 
agree.”  These  items  addressed  concepts  related  to 
preparedness  (e.g.  “The  online  courses  better  pre¬ 
pared  me  for  these  joint  operational  planning 


discussions”),  coordination  (e.g.,  “The  academics 
included  content  related  to  multinational  coordi¬ 
nation”),  value  (e.g.,  “Overall,  the  online  courses 
were  a  valuable  supplement  to  the  academics”),  and 
effectiveness  (e.g.,  “Overall,  I  learned  new  joint  op¬ 
erating  approaches  from  the  online  courses”). 

The  questionnaire  also  asked  participants  to 
respond  to  three  open-ended  questions  to  assess 
the  perceived  usefulness  of  the  online  courses,  nec¬ 
essary  improvements  for  online  training,  and  any 
additional  comments  relevant  to  the  online  courses 
associated  with  PANAMAX  or  e-learning  on  JKO. 
Individual  questions  related  to  each  concept  were 
aggregated  and  reported  on  a  5-point  scale,  and  the 
open-ended  questions  were  assessed  for  qualitative 
trends. 

Similarly,  the  control  participants’  reactions 
questionnaire  asked  them  to  rate  seven  items  on 
a  5 -point  scale,  ranging  from  “strongly  disagree” 
to  “strongly  agree.”  These  items  addressed  con¬ 
cepts  related  to  preparedness  (e.g.  “I  felt  like  I  could 
meaningfully  contribute  to  the  discussions  during 
academics”)  and  potential  value  of  an  online  pre¬ 
course  (e.g.,  “I  would  recommend  that  staff  com¬ 
plete  basic  information  online  before  the  academ¬ 
ics”).  The  questionnaire  also  provided  participants 
with  five  open-ended  questions  to  assess  perceived 
usefulness  of  an  online  course  and  describe  the  pre¬ 
paredness  of  other  staff  members.  Individual  ques¬ 
tions  related  to  each  concept  were  aggregated  and 
reported  on  a  5 -point  scale,  and  the  open-ended 
questions  were  assessed  for  qualitative  trends. 

Post-exercise  reactions.  All  participants  received 
similar,  but  slightly  different,  reaction  question¬ 
naires  at  time  04,  in  order  to  collect  feedback  about 
their  experience  with  the  PMX12  exercise. 

Experimental  participants  completed  a  ver¬ 
sion  of  the  survey  that  included  items  primar¬ 
ily  focused  on  how  effectively  they  believed  the 
pre-exercise  online  courses  had  prepared  them  for 
the  exercise.  The  questionnaire  asked  participants 
to  rate  15  items  on  a  5 -point  scale,  ranging  from 
“strongly  disagree”  to  “strongly  agree.”  These  items 
addressed  preparedness  (e.g.,  “The  online  courses 
better  prepared  me  for  Joint  Operational  Plan- 
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ning  in  PANAMAX”),  participation  (e.g.,  “Dur¬ 
ing  PANAMAX,  I  personally  participated  in  in- 
terorganizational  coordination”),  effectiveness  (e.g., 
“Overall,  I  learned  new  Joint  operating  approaches 
from  the  online  courses”),  and  value  (e.g.,  I  got 
more  out  of  PANAMAX  because  I  completed  the 
online  courses”). 

The  questionnaire  also  included  six  open-end¬ 
ed  questions,  prompting  participants  to  describe 
all  of  the  pre-exercise  activities  they  completed, 
the  value  of  those  activities,  recommend  alterna¬ 
tive  activities,  how  helpful  the  online  courses  were 
in  completing  duties  during  PANAMAX,  recom¬ 
mendations  for  online  training,  and  additional 
comments.  Individual  responses  related  to  each 
concept  were  aggregated  and  reported  on  a  5 -point 
scale,  and  the  open-ended  questions  were  assessed 
for  qualitative  trends. 

Control  participants  also  completed  a  version 
of  the  post-exercise  reactions  survey  regarding  the 
perceived  effectiveness  of  the  standard  pre-exercise 
activities.  The  questionnaire  provided  participants 
with  four  open-ended  questions,  asking  them  to 
describe  what  pre-exercise  activities  were  com¬ 
pleted,  the  value  of  those  activities,  recommend 
alternative  activities,  and  provide  any  additional 
comments  relevant  to  pre-PANAMAX  training  in 
general.  All  the  questions  were  assessed  for  qualita¬ 
tive  trends. 

Course  completion  information.  In  addition 
to  collecting  participant  questionnaires,  we  also 
logged  technical  data  on  the  experimental  partici¬ 
pants’  CoL  course  completion  rates.  This  techni¬ 
cal  information  included  a  count  of  the  number  of 
participants  who  completed  the  CoL  courses,  the 
number  of  people  who  were  assigned  to  take  the 
CoL  courses,  and  when  participants  completed  the 
courses.  We  also  logged  and  analyzed  efficiency 
factors  related  to  course  completion,  including  pre¬ 
test  pass/fail  rates,  number  of  post-test  re-takes  be¬ 
fore  passing,  and  completion  time. 

Participants 

US  joint  military  personnel  involved  in  the  PMX12 


exercise  were  recruited  to  participate  in  this  study 
(iV=  196).  Just  over  100  participants  ( n  =  106)  com¬ 
pleted  the  experimental  CoL  and  some  portion  of 
the  experimental  battery.  The  others  ( n  =  90)  par¬ 
ticipated  as  a  control  group.  However,  not  all  par¬ 
ticipants  completed  all  measurements;  consequent¬ 
ly,  the  numbers  per  analysis  vary. 

Because  participants  were  active  and  reserve 
military  members,  recruitment  was  coordinated 
with  their  leadership  at  USSOUTHCOM  head¬ 
quarters  (Doral,  FL)  and  US  Naval  Forces  South¬ 
ern  Command  (Mayport,  FL).1  These  commands 
each  identified  approximately  50  exercise  partici¬ 
pants,  who  were  then  asked  to  become  the  experi¬ 
mental  group  (although  they  could  opt-out  of  the 
data  collection).  All  other  US  exercise  participants 
were  able  to  participate  as  control  group  members. 

Demographic  data  were  examined  as  pos¬ 
sible  correlates  and / or  covariates  to  knowledge  test, 
joint  mindset,  and  reaction  survey  scores.  Signifi¬ 
cant  demographic  differences  did  exist  based  on 
rank  and  previous  service  designation  for  several  of 
the  questionnaires.  Therefore,  these  variables  were 
controlled  for  in  analyses,  where  appropriate  (in¬ 
dicated  in  each  analysis  description).  The  data  were 
also  checked  for  significant  outliers  on  each  out¬ 
come  variable,  and  statistically  meaningful  outlying 
responses  (more  than  two  standard  deviations  from 
the  mean)  were  discarded.  An  overview  of  relevant 
demographic  variables  is  presented  in  Table  4.2. 

CoL  VI.O  Beta  Test  Components 

Experimental  group  participants  completed  three 
online  CoL  courses  prior  to  PMX12: 

•  JFC  100  Joint  Operations  Planning 

•  JFC  100  Interorganizational/Multinational 

•  JFC  200  Interorganizational  Coordination 


1  USSOUTHCOM,  located  in  Doral,  Florida  (i.e.,  Miami),  is 
the  is  one  of  nine  unified  Combatant  Commands  (CCMDs)  in 
the  Department  of  Defense.  SOUTHCOM  headquarters  supports 
Tier  1  level  decision-making.  Naval  Forces  Southern  Command 
(NAVSO),  located  at  Naval  Station  Mayport,  is  the  Naval  element 
of  SOUTHCOM,  and  it  performs  Tier  2  level  decision-making 
under  the  direction  of  SOUTHCOM  leadership.  During  PMX12, 
NAVSO  also  hosted  the  Multinational  Forces  South  (MNFS)  com¬ 
ponent  of  the  exercise. 
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These  particular  courses  were  selected  by  US- 
SOUTHCOM  leadership  because  of  their  rel¬ 
evance  for  the  exercise.  Beta-test  modules  became 
available  online  in  time  to  support  USSOUTH- 
COM  and  Multinational  Force  South  (MNFS) 
pre-exercise  academics  as  well  as  the  collective 
PANAMAX  exercise,  itself. 

Results 

Course  Completion  Data 

Technical  information  regarding  course  comple¬ 
tion  was  logged  and  output  by  the  JKO  learning 
management  system.  This  information  included  a 
count  of  the  number  of  participants  who  complet¬ 
ed  the  CoL  and  the  dates  each  participant  began 
and  completed  the  modules.  The  digital  log  also 
reported  efficiency  factors,  including  pre-test  pass 
and  fail  rates  and  participants’  completion  times. 
Table  4.3  provides  an  overview  of  the  logged  data. 

Content  Knowledge  Test 

Significant  differences  were  found  on  the  knowl¬ 
edge  test  based  on  rank  (categorized  into  lowest, 
mid,  and  high)  and  Service  (US  Air  Force,  US 
Navy,  US  Marine  Corps,  and  US  Army).  Therefore, 
all  analyses  performed  with  the  knowledge  test 
controlled  for  rank  and  Service  as  covariates.  No 
other  demographic  variables  showed  significant 
differences. 

Outcomes  from  a  univariate  between-subjects 
ANCOVA,  controlling  for  rank  and  Service,  in¬ 
dicate  that  there  was  a  statistically  significant  dif¬ 
ference  between  the  control  {n  =  47,  M=  48.83%) 
and  experimental  ( n  =  57,  M  =  59.09%)  groups  on 
the  knowledge  test,  F(2,  48)  =  8.58,  p  =  .001.  Fig¬ 
ure  4.3  provides  a  graphical  representation  of  scores 
for  the  experimental  and  control  groups  separately. 

Covariates.  As  stated  previously,  a  one-way 
ANOVA  and  accompanying  post-hoc  tests  in¬ 
dicate  statistically  significant  differences  existed 
between  personnel  based  upon  the  Service  desig¬ 
nations.  Specifically,  personnel  from  the  US  Army 


Table  4.2.  Participant  characteristics 


N 

Experimental  Group  =  106 
Control  Group  =  90 

Status 

Active  Duty:  52.94% 

Reserve:  42.65% 

Retired  Military:  1 .47% 

Civilian:  2.94% 

Average  years  served 

Active  Duty:  1 2.43  years 

Reserve:  7.79  years 

Gender 

Male:  80.28% 

Female:  19.72% 

Rank 

Low  (E4-E5,  01-02):  1  1.69% 

Mid  (E6-E7,  03-04):  58.44% 

High  (E8-E9,  05-06):  29.87% 

Service  designation 

US  Navy:  4 1 .67% 

US  Marine  Corps:  6.25% 

US  Army:  25.00% 

US  Air  Force:  27.08% 

Current  joint  status 

Serving  in  joint  billet:  47.83% 

Not  serving  joint  billet:  52. 1 7% 

Time  in  current  billet 

M  =  2.07  years 

Table  4.3.  Technical  information  related  to  CoL  course 
completion  and  efficiency  rates,  aggregated  by  the  JKO  learning 
management  system. 


CoL  Course  Completion  Information 

Completed  CoL  courses 

N  =  106 

Course  completion  dates 

1 5  May  -  1 0  August  20 1 2 

CoL  Course  Efficiency  Information 

Average  pre-test  scores 

M  =  42.03,  SD  =  17.02 

Pre-test  pass/fail  rates 

Pass:  15.09% 

Fail:  84.91% 

Number  of  post-test  re-takes 
before  passing 

M  =  3.84 

Average  completion  time 
for  all  three  CoL  courses, 
including  the  experimental 
apparatus* 

M  =  9.94  hours 

SD  =  70.64  minutes 

*Note:  Many  participants  began  a  CoL  course,  left  their 
e-learning  portal  open,  and  then  completed  the  course  much  later 
(sometimes  as  much  as  10  days  later).  Consequently,  completion 
times  over  20  hours  were  dropped  from  this  calculation.  Similarly, 
completion  rates  less  than  8  minutes  were  dropped;  these  short 
durations  suggest  that  a  participant  merely  looked  at  a  course  and 
returned  later  to  complete  it. 
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performed  significantly  higher  on  the  knowledge 
test  then  personnel  from  US  Navy  and  US  Marine 
Corps,  F(3,  48)  =  2.306,/)  =  .019  and  p  =  .022,  re¬ 
spectively,  and  US  Air  Force  personnel  performed 
the  poorest  (see  Figure  4.4). 

A  one-way  ANOVA  and  accompanying  post- 
hoc  tests  also  indicate  significant  differences  be¬ 
tween  the  lowest  ranking  participants  and  the  mid 
and  highest  ranking  participants  on  the  knowl¬ 
edge  test.  The  mid  (M=  53.01%)  and  highest 
(M=  54.7%)  ranking  participants  performed  sig¬ 
nificantly  higher  on  the  test  than  the  lowest  rank¬ 
ing  participants  (M=  43.52%),  F(2,  48)  =  2.850, 
p  =  .043  (low  vs.  mid)  and  p  =  .017  (low  vs.  high). 
Figure  4.5  provides  a  graphical  representation  of 
knowledge  test  scores  for  the  three  rank  categories. 

Opinions  Towards  Previous  e-Learning  Survey 

All  participants  completed  a  survey  about  their 
opinions  towards  military  e-learning,  in  general. 
The  mean  and  standard  deviation  for  the  entire 
survey  were  calculated:  M  =  2.87,  SD  =  0.93.  Next, 
values  for  each  informal  subscale  were  calculated. 
A  priori,  this  survey  was  subdivided  into  questions 
that  targeted  efficiency  («  =  4),  relevance  {n  =  3), 
effectiveness  {n  =  6),  and  satisfaction  (n  =  3).  Re¬ 
sponses  from  each  reaction  area  were  aggregated 
across  all  participants  and  reported  as  mean  scores. 
No  significant  differences  were  noted  between  the 
control  and  experimental  group  participants,  or  for 
any  demographic  variable  (see  Table  4.4). 

Post-CoL  Reactions  Survey 

After  completing  the  online  CoL  courses,  experi¬ 
mental  participants  responded  to  a  reactions  survey. 
The  mean  and  standard  deviation  were  determined 
for  the  overall  survey:  M  =  3.24,  SD  =  0.88.  Next, 
as  with  the  opinions  towards  previous  e-learning 
survey,  the  subscales  were  analyzed.2 The  post-CoL 
learning  reactions  survey  targeted  each  one  of  the 


2  Although  the  opinions  towards  previous  e-learning  and  post- 
CoL  reactions  surveys  addressed  the  same  constructs,  because  of 
their  different  delivery  contexts,  not  all  items  on  these  apparatus 
were  identical.  See  the  Appendix  to  review  specific  differences  in 
the  wording  on  the  two  apparatus. 


Knowledge  Test  Scores  by  Group 

55% 

50% 

45% 

40% 

35% 

30% 

Experimental  Control 

Figure  4.3.  Average  scores  for  experimental  and  control  groups 
on  the  knowledge  test  given  at  the  beginning  of  PMX 1 2 


Figure  4.4.  Average  scores  by  military  Service  designations  on 
the  content  knowledge  test,  aggregated  across  experimental  and 
control  groups 


Figure  4.5.  Average  scores  by  rank  categories  on  the  content 
knowledge  test,  aggregated  across  experimental  and  control 
groups 
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Table  4.4.  Responses  from  the  control  and  experimental 
participants’  about  their  previous  e-learning  experiences  (i.e., 
military  e-learning  experiences  not  including  the  CoL  courses). 


Opinions  Towards  Previous  e-Learning  Survey 

1  =  Never,  2  =  Rarely,  3= Sometimes, 

4= Frequently,  5=  Always 

Online  learning  is  Efficient 

M  =  2.76 

Online  learning  is  Relevant 

M  =  3.14 

Online  learning  is  Effective 

M  =  3.03 

1  am  Satisfied  with  online  learning 

M  =  2.39 

Figure  4.6.  Average  response  scores  for  post-online  learning 
reactions  by  rank  category 


same  four  areas  as  the  opinions  towards  previous 
e-learning  questionnaire.  Again,  responses  from 
questions  targeting  each  reaction  area  were  aggre¬ 
gated  across  all  participants  and  reported  as  mean 
scores.  No  significant  differences  were  noted  for 
any  demographic  variable.  Table  4.5  provides  an 
overview  of  average  responses  in  regard  to  post¬ 
online  learning  reactions.  The  post-CoL  reactions 
survey  also  asked  several  additional  questions  relat¬ 
ed  to  the  CoL  modules,  specifically  (see  Table  4.6). 

Covariates.  When  assessing  the  overall  post-CoL 
reactions  survey  score,  one-way  ANOVAs  indicat¬ 
ed  statistically  significant  differences  by  rank  (di¬ 
vided  into  low,  mid,  and  high).  Overall,  the  lower 
the  rank,  the  more  positively  the  participant  rated 
the  course  overall  (F(2,  76)  =  .411,^  <  .05). 

•  Lower  ranks:  M=  3.25 

•  Mid  ranks:  M  =  3.19 

•  Highest  ranks:  M  =  2.86 


Table  4.5.  Responses  from  the  experimental  participations  about 
their  reactions  to  the  CoL  courses,  delivered  immediately  after 
course  completion 


Post-CoL  Reactions  Survey 

1  =  Strongly  disagree;  2  =  Disagree;  3  =  Neutral; 

4  =  Agree;  5  =  Strongly  agree 

Online  learning  was  Efficient 

M  =  3.28 

Online  learning  was  Relevant 

M  =  3.51 

Online  learning  was  Effective 

M  =  3.27 

1  was  Satisfied  with  online  learning 

M  =  2.84 

Comparison  of  Previous  Online  Learning 
Experiences  to  CoL 


Efficiency  Relevance  Effectiveness  Satisfaction 


Figure  4.7.  Scores  on  the  opinions  towards  previous  e-learning 
survey  compared  to  the  post-CoL  reactions  survey 

Figure  4.6  provides  a  graphical  representation  of 
this  difference  between  ranks. 

Comparison  of  Previous  e-Learning  to  CoL 

Paired  samples  /-tests  indicated  statistically  signifi¬ 
cant  differences  between  the  participants’  previous 
online  learning  experiences  and  the  CoL  online 
experience.  The  participants  rated  the  CoL  courses 
more  positively  than  their  previous  experiences  in 
every  area. 


Table  4.6.  Responses  for  the  additional,  CoL-specific  items  in 
the  post-CoL  reactions  survey,  completed  by  the  experimental 
participants  immediately  following  CoL  course  completion 


Additional  CoL  Online  Learning  Experiences 

Where  1  =  Strongly  Disagree;  2  =  Disagree; 

3  =  Neutral;  4  =  Agree;  5  =  Strongly  agree 

The  course  required  too  much  time  to  complete 

M  =  3.51 

The  course  modules  were  easy  to  navigate 

M  =  3.62 

The  course  provided  more  engaging  training  than 
other  forms  of  instruction 

M  =  2.82 
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•  Efficiency:  /( 1,72)  =  4.482,/>  <  .000 

•  Relevance:  /(1,70)  =  3.165,/>  =  .002 

•  Effectiveness:  /(1,72)  =  2.183 ,p  =  .032 

•  Satisfaction:  /(1,72)  =  4.928,^  <  .000 

Figure  4.7  shows  the  average  opinions  towards 
previous  e-learning  survey  as  compared  against  the 
post-CoL  reactions  survey. 

Joint  Mindset  Survey 

When  assessing  overall  joint  mindedness  scores, 
no  statistically  significant  differences  were  found 
between  the  control  (M  =2.14)  and  experimen¬ 
tal  (M  =  2.29)  groups,  based  on  outcomes  from  a 
one-way  ANOVA  (p  >  .05).  Therefore,  we  aggre¬ 
gated  the  joint  mindset  survey  scores  across  groups, 
in  order  to  gain  additional  statistical  power,  and 
re-evaluated  the  scores  for  meaningful  correlates. 
Outcomes  from  an  additional  one-way  ANOVA 
showed  no  statistically  significant  difference  in 
joint  mindset  based  on  participants’  Service  or  rank 
(both^  >  .05). 

Several  interesting  trends,  however,  were  iden¬ 
tified.  First,  overall,  joint  personnel  learn,  think 
about,  or  use  joint  concepts  about  once  a  year 
(M  =  2.23,  where  2  =  yearly,  3  =  monthly).  More 
specifically,  personnel  receive  or  seek  out  knowl¬ 
edge  related  to  joint  principles  a  little  more  than 
once  a  year  (M  =  2.21).  They  take  personal  action 
to  increase  their  joint  mindedness  several  times  a 
year  (M  =  2.42),  and  they  perform  joint-specific 
work  functions  about  once  a  year  (M  =  2.00).  See 
the  Appendix  (“Joint  Mindset  Survey”  on  page 
56)  for  the  complete  listing  of  items  and  out¬ 
come  data  for  this  apparatus. 

Post-Academics  Reactions  Survey 

Because  the  experimental  and  control  groups’ 
questionnaires  differed  for  post-academics  reac¬ 
tions,  they  cannot  be  directly  compared.  However, 
several  concepts  were  similarly  assessed  and  can  be 
reviewed  independently.  Based  on  one-way  ANO- 
VAs,  there  were  no  significant  within-subjects  dif¬ 
ferences  for  any  demographic  variable  for  either 
group  (all p  >  .05). 


Overall,  experimental  participants  agreed  that 
the  online  courses  prepared  them  for  contribut¬ 
ing  to  the  academics  portion  of  the  exercise  (over¬ 
all  M  =  3.89,  where  4  =  agree).  Additionally,  they 
would  recommend  that  staff  complete  the  online 
courses  before  academics  (Question  11;  M=  3.92, 
where  4  =  agree)  and  felt  that  the  courses  blend¬ 
ed  well  with  the  academic  session  (Question  10; 
M  =  4.00,  where  4  =  agree). 

Control  participants  felt  only  moderately  pre¬ 
pared  for,  and  confident  in  contributing  to,  the  aca¬ 
demics  portion  of  the  exercise  (M=  3.27,  where 
3  =  neutral).  Also,  36%  of  the  control  respondents 
agreed  or  strongly  agreed  that  an  online  pre-train¬ 
ing  program  might  have  been  beneficial. 

Post-Exercise  Reactions  Survey 

As  with  the  post- academics  reactions  question¬ 
naires,  the  experimental  and  control  groups’ surveys 
differed  for  post-exercise  reactions,  and  therefore, 
cannot  be  directly  compared.  However,  several 
concepts  were  again  similarly  assessed  and  can  be 
reviewed  independently.  Based  on  one-way  ANO- 
VAs,  there  were  no  significant  within-subjects  dif¬ 
ferences  for  any  demographic  variable  for  either 
group  (all^  >  .05). 

Similar  to  the  post-academics  reactions,  ex¬ 
perimental  participants  felt  that  the  online  cours¬ 
es  prepared  them  for  contributing  to  the  exercise 
(overall  M  =  4.26,  where  4  =  agree  and  5  =  strongly 
agree) .  Additionally,  they  felt  that  the  online  courses 
were  a  valuable  supplement  to  the  exercise  (Ques¬ 
tion  12;  M  =  3.50,  where  4  =  agree)  and  that  the 
courses  were  a  logical  first  step  in  the  PANAMAX 
pre-exercise  work-up  (Question  13;  M=  3.82, 
where  4  =  agree). 

Control  participants  again  felt  moderately 
prepared  for,  and  confident  in  contributing  to,  the 
exercise  (M=  3.21,  where  3  =  Neutral). 

Free-Response  Qualitative  Results 

Several  questionnaires  included  open-ended  items, 
which  provided  opportunities  for  participants  to 
provide  more  detailed  responses  in  addition  to  the 
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Table  4.7a.  In  their  reactions  surveys,  participants  were  prompted  to  provide  free  responses  about  their  e-learning  experiences.  Two 
experimenters  independently  reviewed  and  classified  these  responses  by  their  content.  The  most  frequently  mentioned  complains  (or 
compliments,  see  row  4*)  are  listed  in  the  table  below,  along  with  the  percentage  of  participants  who  mentioned  that  issue. 


Frequently  Mentioned  Issues 

Previous  e-Learning 

Percentage  of  respondents 
(control  and  experimental) 
commenting  on  this  issue 
on  the  opinions  towards 
previous  e-learning  survey 

CoL  Courses 

Percentage  of  respondents 
(experimental)  commenting 
on  this  issue  on  the  post-CoL 
reactions  survey 

Online  courses  had  technology  problems,  e.g.,  logging  in  and  browser  crashing 

35.71% 

21.50% 

Online  courses  took  too  long  to  complete 

30.00% 

7.48% 

Online  courses  were  not  relevant 

22.86% 

19.63% 

Online  courses  were  effective  in  developing  and/or  using  critical  thinking  skills* * 

10.00% 

23.36% 

Online  courses  were  not  personal,  lack  interaction 

27.14% 

9.25% 

scaled  items.  For  each  questionnaire,  two  research¬ 
ers  reviewed  the  data,  categorized  the  most  com¬ 
mon  responses,  and  tallied  the  percentage  of  par¬ 
ticipants  who  provided  feedback  that  aligned  with 
each  general  category.  A  review  of  the  most  preva¬ 
lent  responses  on  each  questionnaire  is  provided 
below;  see  the  Appendix  (“Qualitative  Responses” 
on  page  41)  for  a  complete  listing  of  all  partici¬ 
pant  qualitative  responses. 

Opinions  towards  previous  e-learning  and  post- 
CoL  reactions  surveys.  First,  we  compared  the  ex¬ 
perimental  participants’  qualitative  feedback  on  the 
post-CoL  reactions  survey  to  all  responses  on  the 
previous  e-learning  survey.  Overall,  participants 
consistently  provided  more  positive  feedback  about 
the  CoL  courses  than  about  their  previous  online 
learning  experiences.  Tables  4.7a  and  4.7b  provide 
an  overview  of  the  most  common  types  of  feedback 
across  both  questionnaires  and  the  percentage  of 
participants  who  mentioned  each  issue. 

Post-academics  and  post-exercise  reactions  sur¬ 
veys.  We  assessed  the  post-academics  and  post-ex¬ 
ercise  data  using  the  same  method  as  above.  Again, 
several  feedback  categories  were  prominent,  and 
the  percentage  of  respondents  providing  similar  re¬ 
sponses  was  tallied  for  each.  Tables  4.8a  and  4.8b 
provide  an  overview  of  the  most  common  types  of 
feedback  on  these  questionnaires. 


Table  4.7b.  The  list  below  summarizes  some  of  the  additional 
comments  that  expand  upon,  or  are  in  addition  to,  the  items 
summarized  in  Table  4.7a.  Each  issue  below  was  reported  by  at 
around  20%  of  the  participants.  In  this  table,  the  experimenters 
paraphrased  participants’  comments  except  where  noted  with 
quotation  marks;  those  items  use  one  of  the  participant’s  words. 

Opinions  Towards  Previous  e-Learning  Survey 

•  Access:  CAC  login,  bandwidth  problems 

•  Tedious  or  redundant  (20.00%),  Too  many  courses  (7. 14%), 
Outdated  (7.14%) 

•  Best  courses  for  crit  thinking:  Captivity,  SERE  1 00,  Scenario- 
based,  Info  Assurance 

•  “Just  clicking  through  to  get  it  done,”  “Mostly  multiple  choice 
and  regurgitating  information,”  “Just  checking  memorization 
abilities,”  “Becomes  ‘order  of  the  day’” 

•  "Quizzes  are  not  interaction” 

•  "Extra  practice  is  always  good” 

•  “Needs  to  closely  align  with  exercise  to  be  effective” 

•  Navigation  is  often  confusing 

•  Multiple  choice  often  too  easy,  only  requires  skimming 

•  Many  times,  assessment  consists  of  same  questions  as  every¬ 
one  else,  or  last  year 

•  Often  requires  too  much  mastery  learning 

•  “In  general,  military  online  training  is  a  waste  of  time” 

•  Innovative,  relatable,  relevant,  and  interactive  training  would 
be  better 

Post-CoL  Reactions  Survey 

•  Froze,  inaccessible  login,  timeouts,  next  button  not  working 

•  Too  extensive,  just  want  basics 

•  "Portrays  big  picture” 

•  Pictures,  videos,  real-world  analogies  good 

•  Break  into  smaller  segments 

•  Assessments  asked  for  too  much  mastery  learning 

•  Largely  done  on  personal  time 

•  Decrease  assessments  and  use  simulation-based  exercises  to 
validate  learning 
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Table  4.8a.  In  their  post-academics  and  post-exercise  reactions  surveys,  participants  were  prompted  to  provide  free  responses  about 
whether  pre-training  online  courses  would  have  (control  group)  or  actually  did  (experimental  group)  enhance  their  experiences.  Two 
experimenters  independently  reviewed  and  classified  these  responses  by  their  content.  The  most  frequently  mentioned  responses  are 
listed  in  the  table  below  along  with  the  percentage  of  participants  who  offered  that  comment. 


Frequently  Mentioned  Comments 

Control  Group 

CoL  Experimental 
Group 

Online  pre-training  would  have  been  (or  was)  beneficial 

67.50% 

91.67% 

Participant  felt  prepared  for  academics  or  exercise 

17.50% 

81.79% 

Participant  felt  rushed  to  complete  training  requirements  prior  to  exercise 

68.18% 

71.23% 

Discussion 

This  beta  test  was  conducted  as  part  of  a  larger-scale 
initiative  designed  to  improve  collective  training  by 
using  web-based  and  blended  learning  approaches. 
Data  collection  occurred  during  delivery  of  the 
online  courses,  pre-exercise  academic  sessions  for 
Tiers  I  and  II,  and  at  the  PANAMAX  exercise  in 
Mayport  and  Doral,  FL.  Our  main  goal  for  this  test 
was  to  compare  learning  outcomes  and  feedback 
obtained  from  US  PANAMAX  participants  who 
completed  the  CoL  online  training  beta  modules 
(i.e.,  the  experimental  group)  to  those  participants 
who  did  not  (i.e.,  the  control  group).  This  section  of 
the  report  summarized  the  key  finding,  which  were 
described  statistically,  above. 

Content  Knowledge  Test 

The  post-course/pre-exercise  summative  knowl¬ 
edge  assessment  indicated  significant  differences 
between  participants  categorized  in  three  ways. 
First,  as  can  be  expected,  higher-ranking  partici¬ 
pants  scored  better  than  middle  ranks,  who  scored 
better  than  lower-ranking  participants.  Also,  sig¬ 
nificant  differences  existed  between  participants 
from  different  Service  backgrounds.  Specifically, 
Army  personnel  performed  better  than  Navy  and 
Marine  Corps  participants,  who  performed  better 
than  Air  Force  participants.  The  differences  be¬ 
tween  Services  may  not  seem  as  easily  explained 
as  the  differences  in  rank.  However,  anecdotal  evi¬ 
dence  suggests  that  Army  personnel  have  more  op¬ 
portunities  to  engage  in  joint  operations,  and  that 
much  of  the  joint  doctrine  resembles  Army  doc- 


Table  4.8b.  The  list  below  summarizes  some  of  the  additional 
comments  that  expand  upon,  or  are  in  addition  to,  the  items 
summarized  in  Table  4.7a.  Each  issue  below  was  reported  by  at 
around  20%  of  the  participants.  In  this  table,  the  experimenters 
paraphrased  participants’  comments. 

Common  Control  Participant  Responses 

•  Online  training  could  include  knowledge  of  partner  nations’ 
rank  structure,  skills,  military  customs 

•  Would  have  liked  review  of  vocab,  joint  doctrine 

•  Suggested  topics:  Functional  area-specific,  who’s  who  for 
exercise,  exercise  support  flow 

•  Academic  SMEs  knowledgeable,  but  unable  to  convey  info  to 
wide  audience 

•  Job  relevant  content  might  cause  trainees  to  care  more  about 
CBT 

•  Discrepancies  between  trainee  knowledge  and  skills  nega¬ 
tively  impact  training 

Common  Experimental  Participant  Responses 

•  Some  parts  too  macro,  would  like  more  practical  exercises 
within  program 

•  Helped  understand  planning  process,  applied  joint  ops,  good 
review  of  vocab/joint  concepts 

•  More  real  world  examples,  scenarios,  best  practices,  lessons 
learned 

•  Need  more  ways  to  “show”  doctrine  in  application 


trine.  Consequently,  these  Service  members  are  ex¬ 
posed  to  joint  concepts  earlier  and  more  frequently. 

Most  importantly,  however,  the  experimental 
participants  outperformed  the  control  group  by 
21%  (when  controlling  for  the  effects  of  rank  and 
previous  service  designation),  demonstrating  the 
CoL  as  a  successful  learning  tool. 

Opinions  Towards  Online  Learning  and  Post- 
CoL  Online  Learning  Reactions 

When  asked  about  previous  online  learning  experi¬ 
ences,  all  the  participants  (control  and  experimen- 
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tal  combined)  reported  that  online  curricula  are 
only  sometimes  efficient,  relevant,  effective,  and 
satisfying.  In  comparison,  in  response  to  a  concep¬ 
tually  similar  questionnaire,  the  experimental  par¬ 
ticipants  agreed  that  the  CoL  training  experience 
was  significantly  more  efficient,  relevant,  effective, 
and  satisfying  than  their  previous  experiences. 

However,  the  experimental  participants  also 
specified  that  the  CoL  courses  took  too  much  time 
to  complete,  especially  considering  it  was  necessary 
for  most  participants  to  complete  the  training  off- 
duty.  They  also  indicated  that  the  CoL  courses  were 
not  necessarily  more  interesting  than  other  forms 
of  instruction,  demonstrating  the  opportunity  to 
improve  the  courses  in  following  iteration(s).  These 
opportunities  are  discussed  further  in  the  section 
below,  titled  Suggested  Revisions  to  CoL  VI. 0. 

Attitudes  Towards  Joint  Operations 

All  the  participants  (control  and  experimental) 
were  asked  about  joint  concepts,  skills,  and  train¬ 
ing.  We  termed  this  collectively  as  “joint  mindset.” 
Based  on  the  responses,  it  appears  that  the  person¬ 
nel  might  benefit  from  further  cultivation  of  this 
joint  mindset.  In  general,  personnel  only  think 
about  or  use  joint  concepts  about  once  a  year,  and 
they  receive  or  seek  out  knowledge  related  to  joint 
principles  only  a  little  more  than  once  a  year.  They 
take  personal  action  to  increase  joint  mindedness 
several  times  a  year,  and  they  perform  joint-specific 
work  functions  about  once  a  year.  This  seems  to  re¬ 
flect  the  fact  that  most  of  the  exercise  personnel 
are  reservists,  but  we  believe  that  these  areas  point 
to  opportunities  for  improvement.  In  particular, 
this  highlights  the  need  for  training  interventions 
that  enhance  retention  and  additional  sustainment 
training  opportunities  (such  as  those  provided  by 
the  CoL). 

Post-Academics  and  Post-Exercise  Reactions 

Overall,  the  experimental  participants  confirmed 
the  overarching  goals  for  the  CoL  beta  courses.  The 
participants  agreed  that  the  courses  prepared  them 
for  contributing  to  the  academics  portion  of  the 


exercise,  that  they  would  recommend  that  all  staff 
complete  the  online  courses  before  academics,  and 
that  they  felt  that  the  courses  blended  well  with 
the  academic  session.  In  comparison,  the  control 
participants  felt  moderately  prepared  for,  and  con¬ 
fident  in  contributing  to,  the  academics  portion  of 
the  exercise.  Also,  36%  of  the  control  respondents 
agreed  or  strongly  agreed  that  an  online  pre-course 
program  might  have  been  beneficial. 

Similar  to  the  post-academics  reactions,  ex¬ 
perimental  participants  felt  that  the  online  cours¬ 
es  prepared  them  for  contributing  to  the  exercise 
and  reported  that  the  online  courses  were  a  valu¬ 
able  supplement  to  the  exercise.  They  also  indicat¬ 
ed  that  the  courses  were  a  logical  first  step  in  the 
PANAMAX  pre-exercise  work-up.  Alternatively, 
the  control  participants  again  felt  only  moderately 
prepared  for,  and  confident  in  contributing  to,  the 
exercise.  Therefore,  although  the  experimental  par¬ 
ticipants  offered  recommendations  for  improving 
the  CoL  courses,  the  results  indicate  overall  suc¬ 
cess  in  providing  supplemental  preparation  for  the 
PANAMAX  exercise  and  associated  pre-exercise 
training  activities. 

Suggested  Revisions 
to  CoL  V 1 .0 

The  following  recommendations  were  compiled 
based  upon  the  participants’  feedback  and  the  ex¬ 
perimenters’  observation  of  participants’  perfor¬ 
mance.  These  comments  only  address  suggested 
revisions  to  the  CoL  self-paced  courses  (i.e.,  the 
product-level),  since  those  were  the  experimental 
component  that  the  participants  completed.  Sum- 
mative  recommendations  (i.e.,  the  system-level) 
for  the  overall  CoL  effort  are  provided  in  the  next 
chapter. 

Learning  Content 

Include  links  to  official  joint  doctrine  and  other 
source  documents.  Joint  personnel  are  encour¬ 
aged  to  have  an  applied  understanding  of  official 
publications.  The  beta  CoL  included  descriptions 
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of  these  publications  but  no  access  to  the  full  docu¬ 
ments.  As  indicated  in  the  reactions  results,  many 
participants  prefer  to  read  the  actual  doctrine  rath¬ 
er  than  just  see  interpreted  paragraphs.  Therefore, 
we  recommend  including  links  within  the  modules 
to  the  full  official  documents,  which  improves  the 
course  content  for  very  little  cost  or  effort. 

Indicate  explicit  linkages  between  course  content 
and  application.  Although  the  beta  CoL  included 
general  explanations  of  content  application  to  joint 
functions,  the  participants  indicated  that  the  cours¬ 
es  should  explain  with  more  specificity  the  impor¬ 
tance  of  the  content.  We  recommend  incorporating 
more  real-world,  applied  examples  throughout  the 
modules,  where  appropriate.  For  instance,  the  sys¬ 
tem  could  say  “if  you  are  working  in  a  J 2  WG,  then 
this  material  will  help  you...”  Simply  identifying 
and  incorporating  these  functional  area-specific 
links  into  each  module  gives  the  courses  a  more 
tailored  and  personal  feel,  thus  increasing  trainees’ 
motivation. 

Adjust  lower-level  courses  to  emphasize  general 
doctrinal  principles  and  include  more  exercise- 
specific  details.  Many  participants  felt  that  the 
beta  CoL  courses  covered  too  much  detail  in  each 
content  component.  These  participants  reported 
their  desire  to  spend  more  time  learning  basic  doc¬ 
trinal  principles  as  well  as  more  exercise-specific 
information,  especially  regarding  the  partner  na¬ 
tions. 

Therefore,  we  recommend  removing  some  of 
the  more  detailed  joint  process  information  from 
the  lower-level  online  courses  and  instead  incorpo¬ 
rating  material  that  supports  exercise  foundations, 
such  as  an  overview  of  the  partner  nations’  mili¬ 
tary  culture  and  logistics.  These  components  could 
include  rank  structure,  skills  related  to  personnel 
classifications,  and  general  military  customs,  as  well 
as  a  who’s  who  for  the  exercise  and  exercise  sup¬ 
port  flow.  Utilizing  the  training  time  to  brief  these 
exercise  basics  could  improve  ramp-up  time  at  the 
beginning  of  an  exercise  and  help  each  participant 
gain  more  value  from  the  training  experience. 

Include  as  much  real-world  content  as  possible. 


Most  commonly,  the  participants  expressed  their 
appreciation  for  interesting  design  and  low-level 
interactive  components  throughout  each  module. 
We  recommend  supporting  students’  interest  in  the 
course  material  by  increasing  the  number  of  rel¬ 
evant  pictures,  videos,  charts,  scenarios,  historical 
best  practices,  and  descriptions  of  lessons  learned. 
This  type  of  content  gives  courses  a  more  tailored, 
personal  feel,  which  leads  to  enhanced  learning, 
and  increased  motivation. 

Online  Assessments 

Offer  summative  testing  at  end  of  each  module, 
rather  than  solely  at  the  end  of  each  course.  Re¬ 
sults  from  the  knowledge  tests  indicated  a  signifi¬ 
cant  increase  in  knowledge  from  pre-  to  post-train¬ 
ing;  however,  there  is  room  for  improvement.  We 
recommend  incorporating  summative  measures 
of  performance  at  the  conclusion  of  each  module, 
thereby  enabling  the  trainees  to  complete  the  over¬ 
all  course  in  more  efficient  individual  units.  Seg¬ 
menting  the  course  to  include  formal  testing  im¬ 
mediately  after  a  concept  is  presented,  rather  than 
presenting  one  lengthy  test  at  the  end  of  the  entire 
course,  helps  to  increase  learning  outcomes  and 
improves  retention  rates. 

Increase  quality  of  formal  assessments  and  in¬ 
clude  simulation-based  validation  exercises.  As 

indicated  in  the  reactions  results,  trainees  felt  that 
the  summative  assessment  items  were  inconsistent. 
They  reported  that  the  question  quality  ranged 
from  being  too  vague  to  too  specific,  and  some 
questions  were  simply  confusing.  Therefore,  we  rec¬ 
ommend  modifying  the  test  bank  of  questions  to 
more  appropriately  and  consistently  reflect  higher- 
level  cognition,  especially  conceptual  and  applied 
knowledge  of  joint  concepts.  We  also  recommend 
the  development  and  inclusion  of  scenario-based 
exercises  with  assessment  capabilities.  Modifica¬ 
tion  and  development  of  improved  assessment 
techniques  directly  support  the  training,  enabling 
the  delivery  of  more  targeted  feedback  and  reme¬ 
diation. 

Increase  quality  and  quantity  of  feedback.  Provid- 
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ing  specific,  tailored  feedback  is  one  of  the  most  ef¬ 
fective  methods  to  increase  trainee  achievement  in 
learning.  Therefore,  the  online  courses  should  uti¬ 
lize  as  many  opportunities  to  provide  feedback  as 
possible.  Specifically,  the  courses  can  be  improved 
by  including  a  discussion  of  each  assessment  ques¬ 
tion,  and  explanations  for  each  answer  choice,  de¬ 
livered  when  the  assessment  is  complete.  This  way, 
the  trainees  can  learn  from  correct  AND  incorrect 
responses,  and  remediate  accordingly. 

Additionally,  on  the  CoL  pre-tests,  we  recom¬ 
mend  modifying  the  feedback  for  correct/incorrect 
items  to  improve  clarity.  Trainees  expressed  the 
need  for  explanations  (in  text)  about  which  mod¬ 
ules  they  will  have  to  complete,  and  why/how  they 
have  to  complete  them  (e.g.,  “you  only  scored  30% 
on  Module  2,  so  you  will  need  to  review  that  sec¬ 
tion  and  then  take  a  different  test  at  the  end,  on 
which  you  will  need  to  score  80%  to  pass”). 

Motivation  to  Use 

Recommend  to  commanders  to  provide  more 
duty  hours  to  complete  assigned  e-learning 
courses.  A  major  complaint  about  any  DoD  com¬ 
puter-based  training  program  is  the  time  (or  lack 
thereof)  allowed  during  duty  hours  to  complete  it. 
Often,  trainees  must  use  personal  time  for  train¬ 
ing  at  home  or  after  hours.  As  expected,  doing  so 
drastically  decreases  motivation  to  learn.  We  rec¬ 
ommend  advising  commanders  to  budget  an  ap¬ 
propriate  amount  of  hours  to  allow  trainees  to 
complete  the  course  while  on  the  clock. 

Relevance 

Maintain  alignment  with  future  training  or 
events,  and  remain  up-to-date  with  official  doc¬ 
trine.  CoL  beta  trainees  were  positive  about  the 
alignment  between  the  online  courses,  the  academ¬ 
ics  sessions,  and  the  exercise.  The  trainees  felt  that 
the  online  courses  were  an  appropriate  precursor 
to  the  later  exercise-related  activities.  Additionally, 


the  trainees  reported  that  the  content  and  mate¬ 
rial  was  up-to-date  with  joint  doctrine  and  current 
practices.  (Simultaneously,  the  participants  com¬ 
plained  that  many  other  DoD  training  programs 
were  outdated  or  not  directly  applicable  to  specific 
functions  or  activities.)  Therefore,  regularly  adapt¬ 
ing  the  content  to  maintain  this  cohesive  alignment 
and  timeliness  is  highly  important  to  the  continued 
success  of  the  CoL  program. 

Usability 

Streamline  assessments  in  CoL  course.  Beta- 
test  participants  reported  technical  problems  with 
the  assessment  portions  of  the  CoL  courses.  The 
number  of  post-test  questions  was  inconsistent 
over  multiple  attempts,  and  hyperlinks  between 
the  module  content  and  assessments  often  failed 
to  work  or  routed  to  the  wrong  page.  Trainees  also 
reported  frustration  with  the  post- test  remediation, 
which  starting  from  the  beginning  of  a  module 
rather  than  directing  back  to  the  specific  content 
associated  with  a  test  item. 

Revise  the  naming  scheme  used  in  the  JKO 
course  catalogue.  Participants  reported  difficulty 
finding  courses  in  the  JKO  course  catalogue,  even 
though  they  received  explicit  directions  on  how  to 
locate  the  classes.  During  Tier  2  delivery  of  the  on¬ 
line  courses,  experimenters  observed  students  re¬ 
ferring  to  succinct,  printed  instructions  but  yet  still 
struggling  to  use  the  course  catalogue.  To  address 
this,  individual  course  names  should  be  refined  to 
help  students  more  easily  locate  them.  Courses  at 
different  levels  (e.g.,  100  and  200)  should  have  dif¬ 
ferent  course  numbers  that  follow  a  pattern  which 
is  logical  to  an  e-learning  novice,  and  course  names 
should  adequately  reflect  their  content.  For  in¬ 
stance,  Module  8  for  JFC  100  is  currently  unrelated 
to  Module  8  for  JFC  200,  but  this  is  not  evident 
from  their  names,  which  the  trainees  found  confus¬ 
ing.  Eliminating  this  confusion  is  simple  and  can 
be  completed  for  very  little  cost. 
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RECOMMENDATIONS 


This  chapter  provides  systemic  recommendations  for  the  future  of 
the  CoL,  as  well  as  the  approach  to  blended  individual/collective 
joint  training.  These  recommendations  are  based  upon  the  results  of 
the  front-end  analysis  (Chapter  1),  spirit  of  the  CoL  conceptual  de¬ 
sign  (Chapter  2),  and  data  collected  from  the  initial  test  and  evalua¬ 
tion  of  CoL  VI. 0  (Chapter  3). 

Recommendations 

1.  Revise  specifications  for  blended  learning 

The  CoL  VI. 0  initiative  documented  valuable  research-based  rec¬ 
ommendations  about  how  to  best  design  e-learning  and  blended 
learning  experiences  (presented  in  the  front-end  analysis  discussed 
in  Chapter  1),  and  CoL  researchers  uncovered  immediate  feedback 
and  recommendations  from  joint  personnel  who  participated  in  the 
CoL  (presented  in  the  experimental  discussion  in  Chapter  3).  Col¬ 
lectively,  these  recommendations  next  need  to  be  formalized  into 
official  instructions  as  well  as  corresponding  performance  checklists 
that  stakeholders,  such  as  ISDs  and  OTs,  can  actively  use. 

2.  Develop  blended  learning-training  packages 

One  of  the  recommendations  described  throughout  this  report  in¬ 
volves  blending  individual  learning  and  collective  training  oppor¬ 
tunities.  To  help  facilitate  this,  curriculum  developers  should  con¬ 
sider  blended  learning-training  packages  for  Joint  Event  Life  Cycle 
(JELC)  exercises.  These  packages  should  include  intentional  link¬ 
ages,  cross  references,  and  blending  guidance  for  OTs. 

Train  OTs  in  blended  learning  instructional  techniques.  In  addi¬ 
tion  to  providing  tools  for  blended  learning,  it  is  important  to  teach 


One  of  the  CoL  beta  test  participants 
offers  recommendations  for  improving  joint 
individual  training  to  Dr.  David  Fautua 


SECTION  SUMMARY 

This  section  offers  ten  high-level 
recommendations  for  the  future 
of  the  CoL,  and  joint  individual 
and  collective  training  in  general. 

These  include  formally  imple¬ 
menting  the  recommendations 
from  Chapters  1-3  of  this  report, 
revising  the  process  for  assess¬ 
ment  design,  enabling  new  tech¬ 
nologies  on  older  DoD  systems, 
and  implementing  enduring  test 
and  evaluation  metrics  for  learn¬ 
ing/training  system  life-cycle. 
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OTs  how  to  make  good  use  of  those  capabilities. 
OTs  must  learn  about  blended  learning  best  prac¬ 
tices,  as  well  as  receive  training  on  how  to  most 
effectively  use  the  CoL  blended  learning-training 
packages. 

3.  Develop  an  automated  data  “dashboard” 

To  further  facilitate  blending,  as  well  as  to  give 
OTs  and  commanders  more  insights  into  person¬ 
nel’s  individual  and  aggregate  readiness  levels,  the 
system  the  CoL  utilizes  should  automatically  pro¬ 
cess  learning  outcome  data  and  present  read-outs 
of  data  in  an  intuitively  understandable  format, 
couched  in  operationally  applicable  terminology. 
More  specifically,  we  recommend  the  development 
of  a  data  “dashboard”  that  automatically  analyzes 
personnel’s  learning  outcomes  (from  the  online  da¬ 
tabase)  and  presents  easily  interpretable,  dynamic, 
and  interactive  visual  depictions  of  the  aggregate  of 
those  learning  outcomes,  as  well  as  usable  quantita¬ 
tive  scores. 

The  dashboard  should  allow  OTs  and  com¬ 
manders  to  easily  view  relevant  performance  and 
reaction  scores.  By  providing  an  overview  of  aggre¬ 
gate  student  performance  to  the  OTs  prior  to  the 
execution  of  JELC  academics,  for  example,  OTs 
can  present  training  that  is  more  tailored  to  the 
training  audience’s  knowledge  level.  Such  tailoring, 
supported  by  the  dashboard,  should  enhance  learn¬ 
ing  effectiveness  (such  as  the  depth  of  learning  that 
can  be  attained)  and  learning  efficiency  (since  less 
time  will  be  spent  covering  basic  information  the 
audience  has  already  mastered  online).  We  antici¬ 
pate  that  trainee  performance  in  future  blended 
learning-training  packages  will  be  strongly  related 
to  the  quantity  and  quality  of  feedback  provided 
during  the  e-learning  portion. 

4.  Improve  assessment  development 

The  value  of  effective  assessments,  along  with  their 
corresponding  feedback,  cannot  be  understated. 
ISDs,  OTs,  and  other  training  stakeholders  recog¬ 
nize  the  importance  of  creating  good  assessments; 
however,  many  times  stakeholders  are  too  busy  to 


develop  those  assessments.  Revised  processes  must 
adequately  support  those  stakeholders  and  con¬ 
tent  owners,  while  ensuring  that  courses  include 
reliable,  valid,  and  diagnostic  assessments  for  both 
formative  and  summative  evaluation  of  learning 
outcomes. 

5.  Implement  adaptive  e-learning 

As  discussed  in  Chapter  1,  tailoring  (also  called 
personalizing  or  adapting)  instructional  mate¬ 
rial  and  delivery  mechanisms  to  better  meet  stu¬ 
dents’  needs  can  have  profound  impacts  on  learn¬ 
ing  effectiveness  and  efficiency,  as  well  as  learner 
satisfaction.  Because  these  benefits  can  be  gained 
without  requiring  a  significant  investment,  adap¬ 
tive  e-learning  can  potentially  yield  high  returns  on 
investment.  Hence,  we  recommend  broader  imple¬ 
mentation  of  adaptive  e-learning  methods  for  the 
CoL.  Below  are  more  specific  recommendations 
for  implementing  adaptive  e-learning  within  the 
joint  learning  continuum. 

Adjust  courses  based  upon  individual  character¬ 
istics.  Simple  mechanisms  can  dynamically  adjust 
course  content,  learning  progression  (i.e.,  the  order 
of  content),  or  instructional  activities  based  upon 
variables,  such  as  rank,  duty  assignment,  or  other 
input  provided  prior  to  beginning  a  course.1  For 
instance,  an  online  course  might  selectively  include 
different  example  situations  based  upon  which  the 
functional  area  to  which  a  student  is  assigned. 

Adjust  courses  based  upon  group  characteristics. 

Similarly,  courses  can  be  tailored  to  a  specific  cohort 
of  students,  such  as  all  of  the  personnel  completing 
a  certain  exercise.  For  instance,  a  different  use-case 
could  be  selected  for  an  online  course  based  upon 
the  themes  associated  with  exercise  that  the  group 
is  attending.  While  this  capability  is  currently  avail¬ 
able  though  JKO,  it  is  not  an  automated  process. 

1  Formally,  this  form  of  adaptation  is  called  macro-adaptation. 
“Macro-adaptive  instructional  decisions  occur  before  training  actu¬ 
ally  begins”  (Shute  et  al.,  2001,  p.  5).  Broad  adjustments  to  the  in¬ 
structional  environment  are  made  based  upon  data  about  the  learner, 
collected  prior  to  an  instructional  session.  Adaptation  occurs  before 
each  block  or  session  of  instruction.  See  VanLehn  (2006)  for  an  in¬ 
troduction  to  macro-  and  micro-adaptation. 
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Adjust  courses  based  upon  test  performance. 

Other  adaptive  mechanisms  can  adjust  e-learning 
content  or  delivery  features  based  upon  in-course 
performance.2  For  example,  if  a  student  scored 
poorly  on  a  formative  assessment,  the  system  could 
deliver  remediation  targeted  towards  his/her  spe¬ 
cific  knowledge  gaps. 

Facilitate  automatic  adaptation.  Ideally,  all  of 
the  adaptive  mechanisms  discussed  in  this  section 
would  occur  dynamically.  The  learning  manage¬ 
ment  system  should  have  the  capacity  to  receive 
relevant  variables  and  automatically  modify  courses 
in  response  to  them.  Standard  server-side  scripting 
and  programing  approaches  can  readily  facilitate 
this;  however,  the  current  learning  management 
system  does  not  presently  support  these  technolo¬ 
gies.  Currently,  when  tailored  courses  are  developed 
(such  as  for  the  CoL  V1.0),  JKO  developers  must 
manually  adjust  existing  course  curricula  and  post 
the  variants  as  unique  course  listings  in  the  JKO 
catalogue. 

Although  implementing  automatic  adaptive 
processes  does  not  necessarily  require  sophisticated 
software  development,  it  may  require  more  exten¬ 
sive  changes  to  e-learning  course  design,  necessi¬ 
tate  additional  content  development,  and  require 
server  upgrades.  Therefore,  before  incorporating 
these  additional  in-course  adaptive  mechanisms, 
we  recommend  that  a  cost-benefit  analysis  be  con¬ 
ducted. 

6.  Utilize  an  upgraded  LCMS 

To  support  the  adaptive  learning  mechanisms  dis¬ 
cussed  in  the  previous  recommendation,  the  CoL’s 
learning  content  management  system  (LCMS) 
must  be  upgraded  beyond  the  scope  of  many  older 
DoD  content  management  systems.  Similar  up¬ 
grades  are  required  to  enable  the  delivery  of  more 


2  Formally,  this  form  of  adaptation  is  called  micro-adaptation.  “Mi¬ 
cro-adaptive  instruction  occurs  during  the  training  process”  (Shute 
et  al.,  2001,  p.  5).  Micro-adaptation  refers  to  on-going  adaptations 
made  during  the  course  of  one  session,  based  upon  performance  or 
behavioral  assessments  (Modritscher,  Garcia-Barrios  &  Giitl,  2004). 
For  instance,  a  micro-adaptive  system  might  provide  feedback  based 
on  student’s  solution  history  (Ong  &  Ramachandran,  2005). 


sophisticated  assessments,  associated  feedback,  and 
the  evaluation  and  display  of  assessment  data  via 
the  data  dashboard  (recommendation  3). 

7.  Increase  e-learning’s  usability  and  reliability 

Trainees  using  DoD  e-learning  systems  complain 
about  the  low  reliability  networks,  browser-soft- 
ware  conflicts,  and  other  usability  issues.  All  of 
these  technical  issues  increase  overall  training  time 
and  frustration,  while  decreasing  learning  efficien¬ 
cy,  interest,  and  motivation — potentially  causing 
dramatic  reductions  to  the  return  on  investment. 

In  addition  to  utilizing  a  LCMS  with  expand¬ 
ed  technical  capabilities,  the  CoL’s  learning  man¬ 
agement  system  must  also  have  high  usability  and 
reliability.  To  most  robustly  support  the  joint  com¬ 
munity,  the  CoL’s  LCMS  must  integrate  with  the 
other  Services’learning  management  systems  (e.g., 
so  that  a  single  trainee  does  not  have  to  repeat  the 
same  training  on  multiple  system). 

Redesign  course  navigation  to  be  more  usable. 

In  the  CoL  VI. 0  beta  test,  the  trainees  frequently 
reported  frustration  with  the  usability  of  course 
navigation  buttons,  especially  those  between  dif¬ 
ferent  components  in  a  module.  For  example,  after 
the  introductory  vignette,  students  must  click  the 
arrow  (forward)  button.  This  takes  them  to  the  sec¬ 
ond  page  of  that  module,  which  simply  tells  them 
to  click  on  “next  module”  (i.e.,  that  is  the  only  con¬ 
tent  on  page  2).  Alternatively,  if  a  student  clicks  on 
“next  module”  from  page  1,  without  going  to  page 
2,  then  the  system  directs  that  participant  back  to 
the  beginning  of  the  module  without  additional 
explanation.  During  our  observations  at  the  Tier  2 
PMX  testing,  several  students  could  not  figure  out 
how  to  get  past  this  usability  issue. 

This  recommendation  is  listed  in  this  chapter 
(instead  of  with  the  product-focused  recommen¬ 
dations  of  the  previous  chapter)  because  it  reflects 
a  systemic  issue.  The  usability  of  military  online 
learning  experiences,  in  general,  should  be  en¬ 
hanced.  Revising  the  overall  navigation  to  increase 
course  usability,  for  instance,  is  a  relatively  low- 
cost,  low-effort  approach  to  improve  the  course- 
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ware,  and  in  turn  increase  instructional  efficiency 
and  student  satisfaction. 

Modernized  relevant  e-learning  portals.  On 

their  reactions  surveys,  beta-test  participants  of¬ 
fered  some  recommendations  about  improving 
the  web -based  portals  that  they  used  to  reach  the 
e-learning  courses.  Simply  altering  small  details 
in  most  web  portals’  graphical  user  interfaces  will 
make  them  easier  to  use.  For  example,  in  a  e-learn- 
ing  portal’s  course  catalogue,  revising  course  names, 
numbers,  and  organization  to  be  more  readable  to 
the  students  can  significantly  improve  the  usability. 
(Names  that  makes  sense  to  the  internal  develop¬ 
ment  team  may  be  confusing  to  the  training  audi¬ 
ence  and  make  it  more  difficult  for  them  to  find  the 
classes  they  desire.)  Consequently,  we  recommend 
conducting  a  usability  review  of  portals  relevant  to 
the  CoL  and  then  redesigning  their  interfaces  to 
enhance  the  usability  for  trainees. 

Diagnose  IT  malfunctions  and  define  solutions 
for  improvement.  As  implied  in  the  trainee  reac¬ 
tions  section  (Chapter  3)  and  observed  anecdotally 
by  the  researchers,  network  and  server  problems 
are  a  significant  concern  in  e-learning  systems.  Ac¬ 
cording  to  the  beta-test  participants,  throughout 
their  careers  many  of  them  had  had  to  re-take  por¬ 
tions  of  courses  because  of  server  crashes,  network 
connectivity  issues,  or  other  technology  malfunc¬ 
tions.  Other  IT  struggles  reported  by  beta-test  par¬ 
ticipants  included  trouble  logging  in  to  the  courses 
from  home  or  when  using  a  password  rather  than 
a  CAC;  cryptic  instructions  on  hardware  and  soft¬ 
ware  requirements;  and  even  challenges  when  at¬ 
tempting  to  print  course  completion  certificates. 

Enhance  interoperability  among  Services’  learn¬ 
ing  management  systems.  Several  students  report¬ 
ed  experiencing  usability  issues  created  by  gaps  be¬ 
tween  the  various  military  e-learning  portals,  such 
as  JKO,  Army  Knowledge  Online,  or  MarineNet. 
For  instance,  one  senior  enlisted  staff  member 


explained  that  he  was  required  to  complete  the 
10-hour  SERE  course  twice  in  the  span  of  a  few 
weeks,  once  on  JKO  and  once  on  Navy  Knowledge 
Online.  Because  the  two  databases  did  not  effec¬ 
tively  share  course  completion  information,  he  was 
unable  to  get  Navy  credit  for  the  joint  version  of 
the  course  and  vice  versa — even  though  both  sys¬ 
tems  used  the  same  course  content. 

8.  Enhance  course  content  review  processes 

A  more  robust  process  for  reviewing  e-learning,  or 
JOCSIM  scenario,  content  and  delivery  tactics  is 
needed  to  ensure  all  CoL  courses  meet  a  consis¬ 
tently  high  level  of  quality.  To  facilitate  this,  we  rec¬ 
ommend  incorporating  OTs  into  the  current  con¬ 
tent  review  process  and  incorporating  additional 
review  processes  to  ensure  that  courses  effectively 
incorporate  the  new  CoL-based  design  specifica¬ 
tions  (without  stifling  ISDs’ creativity). 

9.  Develop  meaningful  system  metrics 

As  Joint  Training  adopts  these  recommendations 
and  moves  into  its  “next  evolution,”  it  is  prudent  to 
develop  associated  measures  of  performance,  mea¬ 
sures  of  effectiveness,  and  return-on-investment 
criteria  for  these  activities.  W  recommend  devel¬ 
oping  enduring  training  system  metrics,  as  well  as 
associated  testing  and  continuous  improvement 
policies. 

1 0.  Test  and  evaluate  the  system 

Finally,  as  good  system  engineering  practice  de¬ 
mands,  system  enhancements  should  be  iteratively 
tested  and  evaluated.  As  new  CoL  components 
are  developed,  those  features  should  be  evaluated 
against  the  CoL  VI. 0  beta-test  data.  Also,  a  peri¬ 
odic  (e.g.,  annual)  test  and  evaluation  plan  should 
be  implemented  to  collect  ongoing  data  that  ad¬ 
dresses  the  system-level  metrics  (recommendation 
9).  This  output  will  support  continuous  improve¬ 
ment. 


<<  UNCLASSIFIED>> 


CHAPTER 


APPENDIX 


Appendix  Contents 

“Acknowledgements”  on  page  40 
“Qualitative  Responses”  on  page  4 1 
“Joint  Mindset  Survey”  on  page  56 

“Stakeholder  Knowledge  Elicitation  Semi-Structure  Interview”  on  page  57 


<<  UNCLASSIFIED>> 


Continuum  of  eLearning  |  Appendix 


40 


Acknowledgements 


This  section  includes  acknowledgements  from  the  CoL  experimental  report  authors.  Many  people  helped 
make  the  CoL  and  associated  experimentation  a  success,  and  we  wish  to  highlight  some  of  those  key  per¬ 
sonnel,  here. 


First,  we  would  like  to  acknowledge  the  support  of 
the  Joint  Training  observer-trainers  (OTs).  During 
PANAMAX  2012,  these  dedicated  men  and  wom¬ 
en  made  time  during  their  hectic  training  schedules 
to  help  the  CoL  research  team  integrate  seamlessly 
into  the  exercise,  give  us  insights  into  joint  training 
processes,  and  generally  act  as  subject-matter  ex¬ 
perts  for  our  team.  In  particular,  the  help  of  LtCol 
Matt  Deller  (USAF)  and  LtCol  Jeffrey  Gardner 
(USA,  Ret.),  who  were  assigned  to  the  CoL  effort, 
cannot  be  understated.  Similarly,  the  continued 
support  from  the  Joint  Training  division,  including 
COL  Hall,  COL  Parrington,  and  Mr.  Jay  Reist  was 
critical  to  this  endeavor. 

We  also  wish  to  acknowledge  all  of  the  people  at 
Joint  Knowledge  Online  for  making  this  research 
and  process  development  possible.  JKO  built  the 
beta- test  courses,  and  they  continued  to  work  with 
us  throughout  to  refine  both  the  products  and 
development  processes.  In  particular,  we  wish  to 


thank:  Mr.  Joe  Camacho  and  Mr.  Marty  Vozzo 
for  programmatic  support;  Mr.  Mike  Barnum  and 
Mr.  Roy  “Lanny”  White  for  leading  course  devel¬ 
opment  and  coordination  throughout  the  project; 
Ms.  Cattina  Holt,  Mr.  Jarvis  Rathbone,  Ms.  Jen¬ 
nifer  Hanshaw,  Mr.  Rob  Ibarra,  and  Mr.  Damien 
Sanchez  for  their  tireless  effort  as  the  ISDs  who 
built  these  prototypical  courses. 

Many  thanks  to  Generals|C^  [array  and  Becker  for 
their  guiding  leadership  and  support. 

Finally,  we  would  like  to  thank  both  the  SOUTH- 
COM  and  NAVSOUTH  staff  for  their  willingness 
to  provide  the  critical  feedback  and  overwhelming 
support  for  these  beta- test  courses.  Their  leadership 
during  the  project,  Mr.  Bill  Knightly,  Mr.  Herb 
Warden,  COL  Dallas,  CDR  Mershell  and  CDR 
Richardson,  were  critical  to  helping  us  achieve  our 
goals. 


<<  UNCLASSIFIED>> 


Continuum  of  eLearning  |  Appendix 


41 


Qualitative  Responses 


This  section  lists  the  participants’  raw  free-response  reactions,  which  were  summarized  during  data 
analysis  and  described  in  Chapter  3.  These  responses  were  collected  via  the  Opinions  Towards  Previous 
e-Learning  Survey,  Post-CoL  Reactions  Survey,  Post- Academics  Reactions  Survey,  and  Post-Exercise 
Reactions  Survey  The  results  are  unedited  except  for  obvious  spelling  and  minor  grammar  issues,  to  aid 
readability.  No  responses  were  cut  from  the  list  or  otherwise  edited. 


Previous  Online 
Learning  Experiences 
(All  participants) 

In  your  opinion,  what  are  the  top 

three  gaps/problems  with  military 

online  learning? 

•  1.  Access  (passwords).  2.  Length  of 
training.  3.  Documenting/confirmation 
of  completion 

•  1.  Access  2.  Too  long  3.  Too  many-ev¬ 
eryday  it  “feels”  like  there  is  yet  another 
CBT  that  you  are  mandated  to  complete 

•  1.  Locating  course  on  JKO/no  ease  of 
navigation  2.  Website  crashing  frequent¬ 
ly!  3.  Website  not  tracking  course  prog¬ 
ress  accurately! 

•  1.  It’s  online  2.  Most  training  is  multiple 
choice  (people  guess)  3.  It  puts  people  to 
sleep 

•  1.  Can  be  tedious/monotonous  2.  Time 
allocation  3.  Bandwidth  intensity 

•  1.  Impersonal  2.  CAC  reader  dependent 
3.  Lacks  the  personal  experiences  an  in¬ 
structor  would  provide. 

•  1.  Lack  of  student  interaction/sharing 
to  assist  learning  2.  Students  sometimes 
lack  discipline  to  sit  and  take  an  online 
course  3.  Propensity  for  courses  to  be 
taken  in  one  sitting  regardless  of  time 

•  1 .  Overload  in  information  especially  in 
doctrines  2.  Target  audience  recognition 
3.  Presentation  format  and  user  friendli¬ 
ness 

•  1 .  Repetition.  2.  Buggy  software 

•  1 .  Ability  to  look  in  -  connectivity  2.  Too 
many  courses  to  take 

•  1.  Not  personal.  2.  Problems  with  JKO 
and  AKO.  3.  No  context 

•  1.  Inaccessibility  from  home  due  to  cac 
enabled  software.  2.  Not  working  ever 

•  1 .  Volume  of  users  during  the  drill  week¬ 
end  slows  the  servers  down  2.  Some  are 


extremely  long  in  time  to  complete  3. 
Using  the  search  function  doesn’t  popu¬ 
late  if  the  use  the  name  of  the  course  in¬ 
stead  of  the  course  code 

•  1.  Scope  -  online  courses  going  before 
rational  training  development  (i.e.  CBT 
to  train  sailors  how  to  swab  deck)  2. 
Learner  -  size,  content,  length  -  too  long, 
too  much  bandwidth.  Give  exposure, 
that’s  it.  3.  Disconnect  between  users/ 
supervisors  and  developers 

•  1.  Time,  is  there  enough  time,  is  this  a 
priority?  2.  Balancing  content  (is  there 
too  much  content  to  retain)  3.  General 
knowledge  vs.  learning  processes 

•  1 .  Access  2.  Bandwidth 

•  1.  Irrelevant  2.  Tedious  3.  Inaccessible 

•  1.  Too  many!  No  time  or  little  time  2. 
Difficulty  logging  on,  slow  due  to  down 
bandwidth  3.  Outdated/same  (redundant) 

•  1.  Relevance.  2.  Impersonal.  3.  Outdated 

•  1.  Impersonal,  does  not  connect  to  stu¬ 
dent  2.  Too  long  and  slow  -  lose  attention 
of  the  student  3.  Does  not  feel  real,  stu¬ 
dents  feel  being  mandatory  “check  in  the 
box”  training  does  not  connect  them  and 
wastes  their  time 

•  1 .  Not  enough  time  in  workday.  2.  Imper¬ 
sonal.  3.  Too  long 

•  1 .  Online  training  =  taking  PPTs  and  put¬ 
ting  them  into  a  CBT  2.  Online  content  is 
most  effective  when  interactive.  3.  Inter¬ 
active  =  numerous  quizzes. 

•  1.  Access  connectivity  2.  Access  Con¬ 
nectivity  3.  Access  Connectivity 

•  1.  Access  =  CAC  enabled  doesn’t  allow 
work  from  home  2.  Slow  processing 
=  waiting  20-30  seconds  for  next  slide 
(sometimes  longer) 

•  1.  Repetitive  2.  Sometimes  they’re  too 
long. 

•  1.  Most  aren’t  challenging  2.  Too  much 
multiple  choice  3.  Redundant  early  train¬ 
ing;  easy  to  remember  answer. 


•  1.  Poor  “corns”,  crashes,  no  earphones 
provided,  volume  isn’t  working.  2.  Little 
assistance  is  given  in  NOSCs 

•  1.  Not  enough  time  to  complete  2.  Bor¬ 
ing/repetitive  information  3.  No  Interac¬ 
tion 

•  1.  Bandwidth.  2.  Antiquated.  3.  Repeti¬ 
tive 

•  1.  Not  navigational  user  friendly  2.  Site 
goes  down  or  freezes  on  you  3.  Informa¬ 
tion  takes  too  long  to  load  and  trying  to 
work  from  home  or  non-military  com¬ 
puter  is  like  pulling  teeth 

•  1 .  Connectivity  2.  Site  down  issues 

•  1.  Technical  -  access  2.  Repetitive  3.  As¬ 
sessments  mostly  require  you  to  regulate 
information 

•  Keeping  student  engaged  -  interested  - 
motivated 

•  1.  Lack  of  connection  with  real  world 
scenarios.  2.  No  proper  guidance.  3.  No 
proper  understanding 

•  1.  Technical  difficulties  2.  Training  too 
long  -  need  to  have  more  building  blocks 
rather  than  “take  all”  attitude  3.  CAC- 
card  dependence  -  disadvantage  for  re¬ 
servists 

•  1 .  Test  questions  do  not  change  between 
iterations,  allowing  individuals  to  simply 
retake  them.  2.  It  is  easy  for  individuals 
to  cheat  in  a  number  of  ways.  3.  Users 
can  often  lose  interest  taking  in  material 
that  they  already  know 

•  1.  Knowing  training  availability.  2.  Bor¬ 
ing  and  impersonal.  3.  Having  the  time 
vs.  day  to  day  work  hours 

•  Coherent  prioritization  of  all  training/ 
learning  requirements  to  include  military 
training  generic... service  and  joint... 
MOS  specific 

•  1 .  Restricted  to  CAC  login 

•  1 .  Too  long.  2.  More  examples  needed.  3. 
More  reward  or  credit  (need  motivation) 

•  1.  Outdated.  2.  Not  related  to  job.  3.  Tech 
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issues 

•  1.  Redundant  .2.  Impersonal.  3.  Fluffed 
up  with  unnecessary  issues 

•  1.  Redundancy.  2.  Soft/hardware  issues 

•  1.  Broadness  of  material.  2.  Length  of 
material  being  covered  (HOURS).  3. 
Training  intervals  (annual) 

•  1.  Online  training  does  not  reflect  real 
world  utilization  (doctrine  vs.  reality).  2. 
Training  does  not  resonate  with  trainee. 
3.  Most  people  view  it  as  necessary  evil 

•  1.  Lack  of  immediate  feedback  when 
questions  arise.  2.  Some  questions  too 
basic 

•  1 .  Only  as  useful  as  the  person  applying 
himself  to  it.  2.  Still  able  to  click  through 
slides  and  muddle  through  test  to  pass 

•  1.  Lack  of  real  world  examples  linked 
to  joint  training,  techniques,  procedures. 
2.  Training  takes  place  outside  regular 
work  hours.  3.  Redundant  training 

•  1.  Too  long.  2.  Not  relevant  -  needs  to 
be  more  specific.  3.  Inability  to  click 
through  pages,  must  wait  for  speaker  to 
stop  talking  before  moving  on  and  some 
people  read  faster  than  speech 

•  1.  Leadership  looks  at  it  as  “additional 
duty”  2.  Becomes  a  task  to  get  a  block 
checked-any  way  possible  3.  Online 
learning  or  online  certificate  printing 

•  Ease  of  use  -  technology  compatibility 

•  1.  Time  consuming.  2.  Long  online 
course 

•  1.  Dry.  2.  Access  internet  issues.  3. 
Length 

•  NOT  enough  time  due  to  normal  duties 

In  your  experience,  how  effectively 
do  online  military  courses  encour¬ 
age  personnel  to  develop  critical 
thinking  skills?  Can  you  provide  a 
personal  example? 

•  Not  very  effective.  Critical  thinking 
training  needs  to  be  in  person  instructor 
guided 

•  Everyone  learns  differently  and  often 
times  online  courses  are  simply  a  box  to 
be  checked.  I  feel  training  event  utilizing 
“live”  training  (e.g.  face  to  face,  Skype, 
DCO,  chat,  etc.)  is  best  and  could  com¬ 
plement  a  CBT. 

•  Somewhat  effective.  See  100.1  required 
application  of  theory 

•  They  are  not  effective  at  this.  Ever. 


•  Somewhat.  This  is  best  leveraged  when 
there  is  a  critical  assessment  that  forces 
the  member  to  think  through  a  problem 

•  Not  much.  There  is  no  one  to  borrow 
ideas/thoughts  off  of 

•  Some  courses  do.  It  depends  on  the  type 
of  information  within  the  course. 

•  I  have  not  seen  an  online  or  instructor 
LEAD  course  that  has  encouraged  criti¬ 
cal  thinking 

•  Almost  never.  It  takes  memorization  to 
pass  the  online  courses  unless  writing  is 
involved  that  is  also  instructor  graded 

•  Don’t  believe  develop  critical  thinking 
skills.  Increase  knowledge  and  develop 
skills  but  encouragement  comes  from 
human  interaction 

•  Rarely,  because  we  are  just  trying  to  get 
through  the  training  because  we  have  to 
get  it  done  on  our  personal  time  vs.  time 
set  aside  during  training  to  complete. 

•  Yes  trafficking  in  person  course  makes 
you  think  about  the  criminal  culture  dis¬ 
played  in  some  countries  in  regards  to 
human  beings 

•  Critical  thinking  is  utilized  to  get  through 
the  training  faster 

•  I  opine  that  online  courses  do  not  en¬ 
courage  critical  thinking.  It  provides 
overviews  on  regulations,  processes,  and 
organizational  structures 

•  Not  at  all 

•  They  don’t  because  of  how  many  we 
have  to  accomplish.  Most  people  just 
click  through  wanting  to  get  it  done. 

•  Rarely  do 

•  To  get  someone  to  think  critically  re¬ 
quires  a  degree  of  “buy  in”  by  the  stu¬ 
dent.  If  they  don’t  connect,  or  even  deal 
well  reading  off  a  computer  screen,  the 
amount  of  critical  thinking  will  suffer 

•  I  don’t  believe  in  online  training  based 

•  IMHO,  critical  thinking  development  re¬ 
quires  true  interactive  events  (not  quiz¬ 
zes/tests).  When  online  courses  resemble 
online  games  then  true  learning  will  oc¬ 
cur. 

•  Not  effective,  click  through  take  test 
complete  objective,  constantly  mandated 
to  take  B.S.  courses  instead  of  complet¬ 
ing  REAL  MISSION 

•  The  info  assurance  CBT  is  well  done, 
although  it  hasn’t  changed  too  much. 
Users  actually  “solve”  problems.  In  con¬ 
trast  the  JTF  101  CBTs  generally  involve 
answering  “recall”  type  multiple  choice 


questions. 

•  None 

•  No 

•  Not  as  effective  as  most  may  think-most 
reservists  rush  through  it  and  struggle 
with  the  courses  that  have  no  relation  to 
what  they  are  doing  in  country 

•  No,  online  courses  do  not  offer  ability 
to  discuss  questions  or  clarifications  on 
points 

•  No.  General  military  training  is  the  same 
every  year,  very  boring 

•  Online  courses  without  hands-on  dem¬ 
onstration  and  performance  is  not  very 
effective. 

•  I  believe  if  one  takes  the  course  serious¬ 
ly,  it  can  be  a  great  learning  experience. 
For  me  I  really  enjoyed  the  new  SERE 
100  online  course.  The  interaction  really 
made  me  think 

•  Very  ineffective 

•  Not  much  critical  thinking... mostly  due 
to  wanting  to  finish  course  as  quickly  as 
possible 

•  I  say  they  don’t.  As  an  educator,  I  see 
more  repetitive  techniques. 

•  I  do  not  see  how  they  build  critical  think¬ 
ing  skills.  Besides,  online  do  not  build 
critical  thinking  skills  in  most  soldiers 

•  Extremely  rarely.  The  military  promotes 
obedience  to  doctrine  and  authority  as  a 
function  of  organizational  culture.  When 
I  have  challenged  convention  military 
thinking  in  the  past,  senior  leaders  who 
disagree  but  cannot  justify  their  stance 
through  thoughtful  response  resort  to 
flouting  rank  as  evidence  of  greater  un¬ 
derstanding  and  experience 

•  No.  Critical  thinking  is  more  effective 
when  in  classroom/group  training  envi¬ 
ronment. 

•  Hard  to  do  critical  thinking  without  dis¬ 
cussion 

•  Depends  on  how  much  the  course  is 
linked  to  a  career  advance  or  skill  set. 
How  the  course  directly  impacts  an  in¬ 
dividual  career 

•  They  are  better  suited  for  new  or  junior 
personnel  who  are  still  learning  basics 

•  Not  very  much 

•  Don’t  think  eleaming  requires  critical 
thinking.  Written  to  be  easily  absorbed 
and  thought  about 

•  Rarely.  Critical  thinking  is  better  fos¬ 
tered  in  a  live,  group,  training  session 
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•  Doesn’t  like  simple  questions  that  are 
fact-checking... likes  those  that  require 
thought  and  analysis 

•  Rarely.  Unless  there  is  a  clear  link  the 
duties  of  the  person  taking  the  training 

•  Scenario  based  course  such  as  the  new 
IA  courses  and  captivity  course 

•  Not  effective  at  all.  Instructor  led  courses 
stimulate  dialogue  and  further  discus¬ 
sion.  Critical  thinking  comes  from  dif¬ 
ferent  points  of  view,  questioning,  doc¬ 
trines,  values,  believes 

•  Not  often 

•  No.  They  are  <expletive  deleted> 

•  Very  effectively,  e.g.  some  of  the  topics 
of  the  online  course  I  covered  then  at  one 
of  my  officer’s  course  but  never  really 
needed  to  apply  it.  During  the  e  course, 
I  had  to  cover  the  topic  along,  indepen¬ 
dent,  and  tested  on  it. 

•  Being  newly  assigned  and  taking  my  1st 
course,  I  have  learned  what  I  don’t  know 
and  that  I  need  to  develop  critical  think¬ 
ing  skills  as  they  relate  to  this  exercise 

•  limited  because  generally  in  person  or 
f2f  discussion  would  encourage  critical 
thinking 

If  you  could  take  a  pre-test,  and 
possible  test-out  of  online  course 
modules,  would  that  change  your 
opinion  of,  or  motivation  for,  online 
learning?  How  so? 

•  NO 

•  To  some  degree  yes  however  I  “feel” 
our  reliance  on  online/CBT  training  has 
decreased  our  knowledge  base.  One 
can  take  and  pass  a  CBT  without  truly 
LEARNING  the  material. 

•  Yes.  It  would  save  valuable  time  if  I  al¬ 
ready  have  the  knowledge  to  pass  the 
course 

•  Avoiding  redundancy  is  always  a  good 
thing 

•  Not  really  with  the  way  online  testing 
works,  knowing  and  reciting  exact  ver¬ 
biage  sometimes  becomes  the  order  of 
the  day,  vice  showing  understanding  of 
key  concepts 

•  No.  I  have  always  perceived  something 
from  any  online  course  I  have  completed. 

•  Yes.  If  I  know  the  material,  I  do  not  want 
to  retake  a  course.  If  a  pre-test  reveals  I 
do  not  know  information,  then  I  am  will¬ 
ing  to  take  it 


•  Yes.  Several  annual  trainings  have  not 
changed  for  years.  Most  people  find 
ways  to  circumvent  the  training  to  take 
the  test. 

•  Yes  if  I  could  get  out  of  the  taking  the 
complete  test  that  would  be  great 

•  If  available  I  would  take  a  pre-test  to  test 
out. 

•  No.  I  personally  learn  more  from  pre  and 
post  but  doesn’t  change  my  opinion 

•  Yes  it  allows  me  to  get  credit  for  train¬ 
ing/experience  that  I  already  have  and 
focuses  on  new  things  I  need  to  leam 

•  Yes  that  would  be  nice  to  implement  in 
a  way  that  if  you  fail  then  you  retake  the 
full  course 

•  Possibly,  but  this  question  does  not  ad¬ 
dress  why  course  has  been  assigned  or 
the  scope  of  knowledge  needed  by  the 
end 

•  I  think  this  would  provide  effective  use 
of  time  using  experience  and  training  to 
test  out 

•  It  allows  you  to  see  the  most  important 
areas  of  that  activity  and  will  allow  you 
to  focus  on  specifics 

•  Yes,  save  time 

•  Perhaps,  would  save  valuable  time  if 
able  to  test  out 

•  The  sooner  I  can  get  that  certificate  the 
better 

•  Possibly,  at  least  I  would  know  the  test 
isn’t  a  total  waste  of  my  time 

•  Yes  it  gives  me  a  chance  to  sit  down  on 
my  own  time  and  review  material 

•  No  because  online  learning  shouldn’t  be 
about  just  learning  concepts  but  of  apply¬ 
ing  them  in  a  distributed/collaborative 
environment.  You  can’t  test  out  of  those 
distributed/ collaborative  experiences . 

•  No,  again  wasting  my  time,  prevents  me 
from  focusing  on  mission 

•  Yes  at  some  level,  there’s  no  incentive 
to  “leam”  anything  since  you  need  to  go 
through  each  slide  anyway.  The  AF  CBT 
on  ChemBio  training  does  this  well. 
If  you  score  high  enough  on  particular 
modules  on  the  pretest,  you  do  not  have 
to  process  through  those  modules. 

•  Yes,  I  would  probably  be  more  motivated 
if  I  knew  I  could  finish  quickly. 

•  No 

•  Possibly  if  allowed 

•  Some,  I  wouldn’t  have  to  sit  through  in¬ 
formation  I  already  know 


•  Yes  I  only  need  to  learn  new  material 

•  Yes,  because  it  measures  the  fact  that  you 
have  current  knowledge  and  comprehen¬ 
sion  of  the  information  and  do  not  need 
to  reinvent  the  wheel. 

•  No 

•  Yes,  this  would  make  it  less  time  con¬ 
suming  so  I  would  be  less  apt  to  put  it  off 

•  It  would  make  it  easier,  people  might  be 
more  motivated  to  get  it  done... specifi¬ 
cally  on  subjects/courses  that  need  to  be 
taken  every  year  or  1 8  months 

•  Yes,  more  time  to  work. 

•  Yes,  I  would  have  a  different  opinion 
because  I  would  be  measured  by  experi¬ 
ence 

•  More  favorable  of  the  course  because  I 
would  not  have  to  waste  time  repeating 
material  that  I  already  know. 

•  Yes.  Give  an  upfront  perspective  of  cur¬ 
rent  knowledge. 

•  Not  really.  Still  takes  time  to  complete 

•  Yes,  it  would  motivate  you  to  pre-read 
and  understand  the  course 

•  Yes  ,  then  we  can  skip  un-necessary 
courses 

•  YES  majority  of  online  courses  are  re¬ 
fresher  training 

•  Yes.  The  ability  to  ensure  member  knows 
topics  and  not  cover  subjects  repeatedly 
would  prevent  spending  work  time  on 
training  that  may  not  be  needed 

•  It  would  be  for  the  better.  I’ve  taken 
some  classes  so  many  times  I  have  the 
answers  memorized 

•  Not  really.  Still  need  to  spend  time  log¬ 
ging  in  and  perfonning  an  online  func¬ 
tion. 

•  Saves  time  if  little  time  is  available 

•  Somewhat 

•  Yes.  I’ve  having  completed  the  same  KO 
training  in  3  courses 

•  Yes.  If  you  know  the  material,  why  sit 
through  the  whole  course?  Only  review 
areas  you  did  not  grasp  in  pre-test  before 
each  section  not  overall  lesson. 

•  No.  What’s  the  point  of  online  learning  if 
I’m  not  “learning” 

•  Sometimes,  because  it  would  reduce  the 
time  read  if  you  knew  the  material. 

•  Yes  - 1  would  be  much  happier  to  do  it 

•  Maybe.  I  feel  that  a  need  for  a  refresher  is 
necessary  especially  when  I  don’t  prac- 


<<  UNCLASSIFIEDS- > 


Continuum  of  eLearning  |  Appendix 


44 


tice  the  topic  of  that  course. 

•  I  don’t  see  myself  “testing  out”  but  I  see 
it  as  a  useful  tool  for  those  who  can  (new 

guy) 

•  Yes  I  would  not  need  to  retake  classes 

If  you  could  practice  some  opera¬ 
tional  tasks  (like  planning)  collec¬ 
tively  in  an  online  simulation  be¬ 
fore  a  staff  exercise,  so  you  think 
that  would  be  useful?  Why  or  why 
not? 

•  Yes,  provided  the  training  was  straight  to 
the  point 

•  Online  simulation  would  be  helpful  how¬ 
ever  I  think  you  NEED  live  interaction. 

•  Yes.  If  someone  has  no  practical  experi¬ 
ence  in  this  area,  any  training  or  practice 
would  help  them  prepare. 

•  Yes,  it  would  allow  a  team  or  individual 
see  the  products  and  process  executed  in¬ 
stead  of  just  reading  about  it 

•  Yes  because  it  could  help  find  some 
faults.  No,  because  I  believe  the  Wargam¬ 
ing  and  human  interaction  is  still  critical 
with  the  invisibles. 

•  Yes.  That  would  make  theory  more  prac¬ 
tical 

•  It  would,  however  the  coordination 
would  be  extremely  complicated. 

•  Yes  but  time  is  always  a  problem 

•  Yes,  maybe,  it  depends  on  how  the  simu¬ 
lation  would  be  presented. 

•  A  collaborative  environment  would  be 
useful,  due  to  the  collective  collaboration 

•  Possibly.  As  it  would  give  me  some  type 
of  expectation  of  the  process,  but  may  be 
problematic  as  it  forms  preconceived  no¬ 
tion  barriers  as  to  how  it  should  VS  how 
it  does  work 

•  Yes  so  when  the  exercise  actually  starts 
you  already  know  the  battle  rhythm  and 
natural  flow  for  the  exercise 

•  Possibly  -  If  it  is  all  the  same  program, 
terms,  flow,  rhythm  etc.  as  the  exercises. 
Then  I  believe  this  would  be  productive. 
What  we  learn  first  we  learn  best  if  we 
learn  wrong  the  retraining  is  extensive 

•  Yes.  PMESII  and  JIPOE  aren’t  practiced 
regularly  to  help  hone  skills  and  see  the 
process  JOPP,  CAP 

•  Yes-practice  became  second  nature 

•  No.  It  would  simply  add  hours  of  use¬ 
less  briefs,  discussions  and  planning  to 


prepare  for  the  “practice”,  furthermore, 
exercises  are  practice  in  themselves. 
Practicing  for  an  exercise  is  practicing 
for  practice 

•  Yes,  I  do  like  the  idea  of  putting  some¬ 
thing  in  practice.  But  also  would  be  de¬ 
pendent  on  the  time  frame 

•  In  theory  yes,  but  in  practice  it  hasn’t 
worked 

•  Possibly.  It  would  have  to  be  well  coor¬ 
dinated  and  during  a  training  stand-down 
day.  Leadership  would  have  to  allow 
time  during  working  to  break  from  nor¬ 
mal  duty. 

•  Yes  more  time,  training 

•  Yes!  Not  only  useful  but  should  be 
mandatory!  Regular  online  simulations 
should  be  part  of  all  career  development 
(cross-service). 

•  No  -  actual  face  to  face,  hands  on  training 
can  not  be  substituted  by  on-line  training 

•  Yes  this  would  address  a  higher  level  of 
learning  than  just  recall. 

•  Yes,  so  you  could  see  the  bigger  picture. 

•  Yes,  you  learn  more 

•  Yes,  interaction  would  foster  better 
learning 

•  Yes  just  as  a  refresher 

•  Yes!  I  love  simulation;  it’s  the  next  best 
thing  to  face  to  face  instruction. 

•  Absolutely  -  a  virtual  training  would  help 
get  the  online  training  in  one’s  mind  be¬ 
fore  attending  an  exercise 

•  It  would  only  be  useful  if  it  was  as 
streamlined  as  possible  so  as  to  not  take 
up  too  much  time 

•  Not  sure 

•  Perhaps.  It  should  include  participation 
from  all  parties  in  order  to  make  it  effec¬ 
tive 

•  I  would  think  that  online  would  be  help¬ 
ful  if  there  was  continuing  access  through 
AKO  without  CAC 

•  No.  with  a  number  of  other  real  world 
responsibilities  (unless  isolated  in  a 
training  environment)  individuals  simply 
would  not  devote  the  necessary  time  or 
attention  to  a  simulation  to  make  it  effec¬ 
tive  training  tool 

•  No,  don’t  have  the  time 

•  For  a  limited  #  of  skill  sets/functions 

•  Yes.  It  would  give  you  an  opportunity  to 
determine  weak  and  strong  areas  collec¬ 
tively  and  who  you  can  go  to  for  help  in 


your  weak  areas 

•  Yes.  But  would  need  to  be  less  than  3 
hours  and  have  good  participation.  For 
a  METOC  officer  there  normally  is  little 
training  that  METOC  can  participate  in 

•  Could  be. . .  .all  situational  dependent 

•  Yes.  Productive,  practical,  set  your  mind¬ 
set  up 

•  Yes.  Practice  is  always  helpful  when 
learning 

•  No.  Because  there  are  too  many  opinions 
about  how  it’s  supposed  to  happen 

•  No.  Planning  simulation  would  be  best 
accomplished  in  a  live,  group  setting 

•  Maybe.  Prefer  F2F 

•  Yes,  if  led  by  a  real  person 

•  Not  sure.  Think  it  might  be  specific 
enough  for  particular  EX  and  not  take 
too  much  time 

•  YES!  I  would  love  to  practice  being  a 
planner.  Learn  better  by  “doing” 

•  Yes  it  would  help  understand  the  big  pic¬ 
ture. 

•  No  -  It  would  destroy  the  joint  process 
-  collaboration  in  person  is  irreplaceable 

•  Yes  if  you  are  not  a  planner  practicing 
will  help  understand 

•  Yes,  any  pre-training  is  useful  if  you 
have  the  time.  Training  points  out  your 
weaknesses  and  better  prepares  you  for 
the  exercise  if  you  are  already  familiar 
with  policy  and  procedures 

•  No  It  would  be  useful  but  I  don’t  think  I 
would  have  the  necessary  time  to  devote 
to  that  task 

Any  other  comments  about  mili¬ 
tary  online  courses  or  recommen¬ 
dations  for  improving  them? 

•  My  experience  with  JKO  has  been  ex¬ 
tremely  frustrating!  I  have  wasted  much 
valuable  time  on  “required”  training 
courses  due  to  failure  of  the  website. 
Time  in  the  military  is  very  valuable,  and 
there  is  none  to  waste.  However,  I  have 
wasted  many  hours  on  the  JKO  course  in 
the  hope  that  they  will  eventually  work 
the  way  they  are  supposed  to! 

•  USN  said  I  have  difficulty  completing 
many  courses  due  to  high  software  re¬ 
quirements.  Example:  Most  still  operate 
IE  6.0  and  minimal  flash  capabilities  and 
limited  bandwidth.  A  local  uploaded  ver¬ 
sion  could  enhance  ability  to  complete 
more  online  courses. 
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•  If  you  put  in  a  time  limit  (i.e.  1  hour 
for  course,  10  min  for  exam)  on  online 
courses,  it  helps  to  simulate  a  traditional 
environment  for  those  not  accustomed  to 
online  learning. 

•  In  general  military  training  is  considered 
a  waste  of  time  as  its  priority  on  scale 
w/  working  hours  is  terrible.  Which  is  a 
cultural  issue,  however  innovative,  inter¬ 
active,  relatable,  relevant  training  espe¬ 
cially  in  their  environment  of  the  target 
audience  would  be  far  more  effective. 

•  Great  way  to  gain  knowledge,  but  not  a 
replacement  for  collaborative  learning 
in  an  interpersonal  environment  “book 
learning”  vs.  “OJT” 

•  If  possible  if  you  can  shorten  some  of  the 
long  courses 

•  1.  Remember  CBT  is  a  supplement  to 
produce  an  asset.  Ask  where  most  in¬ 
tensive  training  should  occur.  Example 
hands  on,  CBT,  class  etc.  2.  Keep  scope. 
Don’t  make  me  do  a  4  hour  course  online 
for  something  I  will  be  doing  for  8  hours 
the  next  day.  3.  Reduce  redundancy. 

•  Continue  with  current  examples,  best 
practices  and  pitfalls  to  keep  training/ 
course  material  relevant  and  effective 

•  Checklist  should  be  created  to  keep  as  a 


reminder. 

•  Eliminate  them 

•  Need  less,  not  more 

•  More  than  30  min  at  a  time  is  too  much. 
Classroom  discussion  is  easier  to  follow 
and  stay  engaged.  I  can  follow  my  col¬ 
lege  profs  for  1 .5  at  a  time,  but  CBTs  lose 
me  after  30  min 

•  Social  networking  games  are  a  model  for 
online  simulations.  Not  only  will  indi¬ 
vidual  learning  occur  but  organizational 
leaming/evolution  can  occur.  Look  at 
disease  research  done  from  the  WoW 
(World  of  Warcraft)  disease/enchantment 
outbreak  as  a  model. 

•  GET  RID  OF  IT!  Promote  classroom 
practical  field  training.  Get  rid  of  the 
idiot  box  computers! 

•  Give  more  bandwidth,  more  storage 
space,  more  time 

•  Add  more  simulation  exercise  and  cut 
out  all  of  the  end  of  module  tests  and  fi¬ 
nal  assessment  tests  “simulation  is  more 
effective  than  fear  of  failing  tests” 

•  Keep  up  the  good  work 

•  Test-out  options  would  be  great 

•  While  online  courses  are  currently  dis¬ 


semination  platforms,  including  a  func¬ 
tion  for  discourse  (chat/forums)  could  go 
greater  lengths  to  engage  users  and  ex¬ 
panding  learning  opportunities 

•  Model  of  progressive/as  you  go  valida¬ 
tion  is  better  than  “test  at  the  end”  meth¬ 
od.  Former  seems  to  be  better  received 
and  more  effective. 

•  NO 

•  Scenarios  where  you  choose  answer 
from  choices  helps  to  relate  material  to 
what  you  read  and  helps  to  reinforce. 
Having  to  obtain  100%  on  tests  is  frus¬ 
trating  and  only  causes  person  to  start 
guessing  or  process  of  eliminate  just  to 
get  through.  *** Anyone  being  assigned 
to  Joint  Comm  should  be  given  a  list  of 
courses  they  should  complete  prior  to  re¬ 
porting  just  like  GSA  assignment,  how¬ 
ever  someone  should  check  for  course 
completion  (unlike  GSA) 

•  YES.  Quit  believing  that  online  learn¬ 
ing  is  the  panacea  for  learning.  Online 
learning  courses  should  last  no  more 
than  1  hour.  Any  longer  than  that  and  it 
bites  into  my  work  day. . . .keeps  me  from 
researching  questions  I  may  have  and 
bores  me. 

•  Somehow  make  them  more  interesting 
they  are  a  little  dry 


Post-CoL  Reactions 
Survey  (Experimental 
Group  Only) 

In  your  opinion,  what  were  the 

least  important  topics  covered  in 

the  online  course? 

•  In  my  opinion,  Lesson  2,  slides  12-14 
were  of  limited  value  due  to  the  loose 
framing  of  the  scenario.  The  recom¬ 
mended  responses  had  little  to  nothing  to 
do  with  the  focus  of  the  rest  of  the  course 
and  therefore  simply  interrupted  the  flow 
of  the  course  material. 

•  The  lessons  were  time  consuming.  The 
test  was  challenging 

•  All  topics  were  helpful 

•  Don’t  know.. .all  seemed  relevant 

•  The  course  should  define  exactly  what  is 
or  is  not  important. 

•  Linear  approach 

•  None 


•  None  noted 

•  JSOPP 

•  JOPES 

•  May  have  been  desperation  to  finish 
module,  but  Lesson  7 

•  Strategic  thinking 

•  I  thought  that  all  subjects  and  topics  dis¬ 
cussed  were  important  to  my  education 
to  be  an  educated  member  of  the  staff. 

•  Crisis  Action  Planning 

•  OPEN 

•  Operational  Art 

•  This  survey. 

•  Operational  Design 

•  Lesson  4:  Joint  Operations  Planning  Pro¬ 
cess  Deliberate  Planning  Contingency 
Plans,  it  was  covered  in  too  much  depth 
and  took  too  long  to  complete. 

•  JOPP  -  An  important  but  very  familiar 
topic. 

•  All  seemed  relevant 


In  your  opinion,  what  were  the 

most  important  topics  covered  in 

the  online  course? 

•  The  JOPP  and  the  need  to  understand  it 
in  order  to  play  well  in  the  Joint  sandbox. 

•  COA,  APEX,  IPR,  Crisis  Action  Plan¬ 
ning,  EXORD,  OPLAN,  OPORD. 

•  In  my  opinion  all,  since  they  were  all 
new  to  me. 

•  All  the  topics  were  important 

•  All  of  them. 

•  APEX,  CAP 

•  This  was  a  good  over  view  of  the  plan¬ 
ning  process. 

•  Initial  planning 

•  CRISIS  ACTION  PLANNING 

•  I  couldn’t  tell  from  the  course.  I  didn’t 
feel  that  anything  was  emphasized  more 
than  anything  else.  This  leads  me  to  say 
that  the  most  important  topics  were  the 
ones  the  questions  were  about. 

•  Structure  of  org 


<<  UNCLASSIFIED>> 


Continuum  of  eLearning  |  Appendix 


46 


•  Roles  of  Commanders  and  Staff 

•  Joint  Planning  Process 

•  Joint  ops  approach 

•  Planning  process 

•  APEX 

•  All  of  it. 

•  Joint  operations  planning 

•  APEX 

•  JOPP  (Lesson  4) 

•  Interaction 

•  Detailed  description  and  teaching  of  the 
Joint  Operational  Planning  Process 

•  Operational  Art  and  Operational  Design 

•  OPEN 

•  Joint  Planning  process  details  and  Orders 

•  CAP,  APEX.  Too  much  info. 

•  Differences  between  APEX  and  JOPES. 

•  All  of  it. 

•  JOPES  and  APEX 

•  Planning  process 

•  Planning 

•  All 

•  APEX  and  its  role  in  planning. 

•  APEX  -  it  is  fairly  new  and  it  is  impor¬ 
tant  to  train  staff  officers  on  it. 

•  The  flow  of  the  planning  process 

Please  describe  any  technical  diffi¬ 
culties  you  experienced  while  com¬ 
pleting  the  online  course 

•  The  multiple  pop-ups  do  provide  one 
more  window  to  have  to  navigate  to  and 
from,  but  otherwise,  no  comments. 

•  Technical  difficulties  -  none. 

•  I  started  on  this  course  over  a  month 
ago-got  off  track  and  am  now  rushing  to 
finish  the  course.  I  thought  the  program 
would  have  done  a  better  job  in  saving 
my  course  progress  so  that  I  would  not 
have  to  start  over  and  over  again. 

•  MINOR  DELAYS  AND  HAD  SYSTEM 
DELETE  OR  HID  TWO  EVALS  LIKE 
THIS  ONE. 

•  Slow  transition  to  next  page. 

•  Timed  out  twice  when  I  had  to  go  do 
other  things. 

•  Worthless 

•  Couldn’t  access  JKO  on  Internet  Ex¬ 
plorer  from  home.  Had  to  use  Internet 


Explorer  here. 

•  Had  to  reload  browser  several  times,  the 
browser  also  froze  and  required  me  to 
close  out  and  log  back  in. 

•  AT  TIMES  THE  PROGRAM  WOULD 
NOT  LET  ME  PROCEED  TO  THE 
NEXT  LESSON 

•  Sometimes  the  internet  was  too  slow. 

•  On  Module  5,  Lesson  4, 1  would  suggest 
that  when  answering  the  6  test  questions, 
the  text  inside  should  be  auto  highlighted 
in  order  to  allow  the  user  to  delete  the 
text  conveniently  and  easily. 

•  Internet  web  site  had  many  freezes. 

•  Kept  having  to  go  through  slides  again 
when  the  link  did  not  take  you  to  the  next 
level. 

•  Site  when  down  occasionally 

•  In  the  Formatted  Assessment  starting 
with  slide  37  the  answers  for  question  5 
&  6  are  the  same 

•  At  one  point  in  Lesson  3, 1  clicked  next 
lesson  and  I  went  from  approximately 
page  25  back  to  page  2.  The  only  way  to 
get  back  was  to  manually  click  forward 
through  each  page. 

•  When  you  select  closed  caption,  the  pop 
up  box  covers  part  of  the  lesson  and 
sometimes  a  picture  or  graphic  that  you 
need  to  view  while  reading  the  CC  box. 

•  None  significant 

•  As  a  reservist  with  JKO  now  requiring 
a  government  network  I  can  no  longer 
work  from  home  on  the  course  this  ex¬ 
tended  the  completion  time  by  at  least  a 
week. 

•  The  vignette  took  5-10  minutes  to  start 
playing,  and  then  froze  after  a  minute  or 
two  of  playback.  Took  another  few  min¬ 
utes  to  restart. 

Any  other  comments  about  this 

online  course  or  recommendations 

for  improvement? 

•  Lesson  1:  Overview  of  the  Joint  Op¬ 
erations  Planning  Process  had  a  graphic 
which  spatially  represented  the  Opera¬ 
tional  above  the  Theater-Strategic  level 
of  war.  It  may  be  small  potatoes,  but  for 
visual  learners  who  are  less  familiar  with 
the  material,  it  may  lead  them  to  initially 
believe  that  the  representation  is  correct 
until  the  text  and  audio  kick  in,  forcing 
them  to  un-learn  the  arrangement  repre¬ 
sented  by  the  graphic. 

•  Having  to  get  100%  on  test  was  frustrat¬ 


ing  and  made  me  just  start  guessing  an¬ 
swers  to  get  through  the  test. 

•  A  lot  of  the  terminology  is  complex  and 
the  courses  should  be  prepared  keep¬ 
ing  in  mind  that  not  all  members  have 
the  same  education,  thus  the  aforemen¬ 
tioned.  Topics  are  good  but  one  must 
keep  that  all  ranks  may  be  assigned  to 
the  exercise  and  must  complete  training, 
so  it  should  be  prepare  or  broken  with  a 
simpler  vocabulary.  Keep  in  mind  your 
audience,  it  is  not  always  going  to  be  the 
more  educated  people. 

•  Make  it  shorter 

•  Overall  I  found  the  course  to  be  chal¬ 
lenging  because  it  was  detailed  infor¬ 
mation  about  the  topic.  I  became  very 
informed  about  the  concepts  involved 
but  will  need  to  engage  in  the  processes 
over  time  to  truly  digest  the  details  of  the 
process. 

•  Redo  it 

•  Take  it  off  the  internet  and  have  an  expe¬ 
rienced  planner  teach  it  to  beginners  in  a 
SOUTHCOM  classroom. 

•  This  module  is  way  too  long.  Not  only 
are  lessons  with  30+  slides  discourag¬ 
ing,  but  they  all  included  “rollover”  and 
“click  here”  portions  that  added  to  the 
overall  content.  By  the  time  I  reached 
the  Post-Assessment,  all  that  information 
had  begun  to  muddle. 

•  Dislike  that  if  one  of  the  mouse-over/ 
click  for  more  info  is  missed  that  you 
can’t  move  on  to  next  topic.  Had  to  take 
Lesson  4  twice  because  of  this.  Too  much 
info  for  one  course,  and  it  took  way  more 
time  than  the  estimate. 

•  Break  up  the  topics  into  shorter  segments, 
not  exceeding  more  than  20+  slides,  with 
small  quizzes  to  test  student  comprehen¬ 
sion  and  emphasize  key  learning  points. 
After  20+  slides,  student  attention  span  is 
diminished  and  learning  reaches  a  dimin¬ 
ishing  point  of  return. 

•  This  survey  is  too  long. 

•  CONSOLIDATE  SURVEYS  WHEN 
TAKEN  MORE  THAN  ONE  COURSE. 
TOO  MUCH  TIME  IS  BEING  WAST¬ 
ED  REPLYING  TO  EACH. 

•  Make  it  go  away 

•  This  training  wasted  a  lot  of  valuable 
time. 

•  I  attended  the  two-week  JECC  Planners 
course  recently.  I  have  worked  in  a  Joint 
environment  since  1987.  SCJ15  Opera¬ 
tions  has  been  extremely  busy  with  real- 
world  Joint  Manning  Document  Sourc- 
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ing.  Our  Section  has  been  undermanned 
with  individuals  in  our  section  TDY  or 
on  leave. 

•  Shorten  into  two  different  modules. 

•  This  particular  module  had  too  much  in¬ 
formation  and  should  be  broken  into  two 
modules 

How  effectively  did  the  online 

course  encourage  you  to  develop 

critical  thinking  skills?  How  so? 

•  No  change. 

•  Effective  by  painting  the  bigger  and 
whole  picture. 

•  It  encourages  more  that  the  normal  for 
me,  to  the  point  that  it  was  annoying  me 
because  a  lot  of  the  terms,  vocabulary 
and  acronyms  I  did  not  know  and  had  to 
look  them  up. 

•  Very  much 

•  Not  at  all,  the  class  is  boring  which 
makes  it  difficult  to  follow. 

•  Minimally 

•  Very  effective 

•  SOMEWHAT,  BY  THINKING  AND 
DETERMINING  WHAT  TASKS 
NEEDED  TO  BE  ACCOMPLISHED  IN 


ORDER  FOR  THE  PROCESS  TO  CON¬ 
TINUE  TOWARD  COMPLETION. 

•  Memorization  -  yes;  critical  thinking  - 
no. 

•  Not  very  effectively,  this  was  more  an 
understanding  of  the  workflow  involved 
in  the  planning  process  than  any  actual 
critical  thinking  skills. 

•  Not  too  much 

•  No  so  much. 

•  Slightly 

•  None 

•  It  didn’t.  I  was  simply  attempting  to  di¬ 
gest  the  concepts  of  the  process. 

•  This  is  crap 

•  Not  very  effectively. 

•  Not  very  effectively,  it  was  mainly  click¬ 
ing  through  PowerPoint  style  lessons  and 
hoping  that  short-term  memory  would 
last  until  the  Post- Test 

•  Not  effective. 

•  Medium 

•  Not  very  effective 

•  Online  “learning”  is  not  conducive  to 
critical  thinking. 


•  I  will  have  to  put  them  into  actual  situa¬ 
tions  to  honestly  answer  the  question. 

•  VERY 

•  Did  not  give  real  world  examples  with 
clearly  linked  planning  info. 

•  Not  at  all.  Critical  thinking  is  not  memo¬ 
rization.  These  courses  do  not  require 
critical  thought.  Check  the  definition  of 
critical  thought. 

•  It  did  not. 

•  Not  much 

•  Effectively 

•  Moderately.  The  lessons  were  very  long 
and  the  overall  module  was  extremely 
long.  It  was  difficult  to  stay  focused. 

•  Well 

•  Very  effective 

•  Marginally,  the  course  could  apply  JOPP 
more  to  real  world  campaigns  and  prob¬ 
lems. 

•  This  is  more  about  learning  the  topic 
than  thinking  critically.  That  will  only 
happen  during  actual  use  of  the  planning 
processes. 

•  Somewhat  effective 


Post-Academics 
Reactions  Survey 
(Experimental  Group) 

Did  the  online  courses  help  you  get 
more  out  of  the  Academics?  If  yes, 
can  you  provide  an  example?  If  no, 
what  would  have  made  the  courses 
more  useful? 

•  Yes.  The  online  courses  focus  more  in 
understanding  the  “how”  of  joint  inter- 
organizational/multinational  relationship 
and  common  effort 

•  Yes,  it  was  a  good  overall  review 

•  Yes,  mostly  vocab 

•  Yes,  I  was  more  familiar  with  some  key 
terms  and  concepts  than  I  would  have 
otherwise  been 

•  Yes  -  courses  provide  examples  and  more 
depth  to  the  topics 

•  Yes.  Mention  of  APEX  planning  system 
was  new  information  that  I  should  have 


been  covered  in  academics 

•  Provided  me  with  terminology  used  dur¬ 
ing  academics 

•  I  did  not  attend  academics,  with  excep¬ 
tion  of  academics  conducted  12  July 

•  NO.  More  useful  if  practice  exercise 
event. . .  for  instance.  Establish  a  joint  tar¬ 
geting  working  group  and  show  every¬ 
one  at  least  what  it  looks  like 

•  Online  courses  very  MACRO.  With  ex¬ 
amples  by  not  focus  on  strategic  or  the¬ 
ater  TTP 

•  YES.  Prepared  the  analysts  mindset  with 
useful  knowledge  to  better  collect  and 
understand  info 

•  YES.  To  better  understand  the  planning 
process 

If  you  could  improve  one  or  two 
aspects  of  online  training,  so  that 
it  better  prepared  staff  for  exer¬ 
cises  like  PANAMAX  and  related 
operational  duties,  what  would  you 
change  and  why? 


•  No  changes 

•  Use  more  “lessons  learned”  and  exam¬ 
ples  than  doctrine 

•  More  details  on  TFFDD  planning  and 
plans  management  board??  And  7  min¬ 
ute  drill.  Use  of  other  COCOM  best  prac¬ 
tices  and  examples 

•  More  specific  scenarios  to  CCDR  AOR 
so  you  can  see  how  the  lesson  “basics” 
fits  into  real  world  scenarios.  Have  a  link 
to  each  COCOM  so  users  can  choose  rel¬ 
evant  one. 

•  One  course  was  very  very  painful  and  not 
very  useful.  At  least  one  course  did  not 
have  practical  applications  /  lessons  to 
apply  real  world  scenarios. 

•  Would  change  the  length.  No  more  than  1 
hour  to  complete.  Keep  it  short 

•  Include  some  of  the  academics  in  the  on¬ 
line  training 

•  All  CBT  should  include  walkthroughs  or 
examples  that  assist  with  real-world  ex¬ 
periences 

•  Length  of  course.  E-course  was  time 
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consuming.  Almost  3  days  to  complete. 
Very  infonnative  though  and  helped  me 
better  understand  planning. 

Any  other  comments  about  the 
pre-PANAMAX  online  courses  or 


eLearning  on  Joint  Knowledge  On¬ 
line,  in  general? 

•  Would  have  been  better  if  put  out  ear¬ 
lier  for  understanding  prior  to  exercise 
RTC  planning.  Obviously  this  won’t  be 
a  problem  for  next  year  :) 


Lessons  ought  to  incorporate  “plan  re¬ 
view”  lessons  from  past  on  similar  mis¬ 
sions.  Those  would  be  more  valuable  to 
theater  planners. 

Where  was  the  SOUTHCOM  participa¬ 
tion  for  it? 


Post-Academics 
Reactions  Survey 
(Control  Group) 

In  your  personal  experience,  how 
well-prepared  are  the  other  indi¬ 
vidual  staff  members  when  they 
arrive  at  the  Academics?  For  in¬ 
stance,  are  some  of  them  rusty? 
Do  they  know  the  current  doc¬ 
trine? 

•  Yes  rusty  or  nonexistent  online  is  not  the 
answer 

•  Difficult  to  tell  at  this  point.  Will  know 
more  as  exercise  goes  on 

•  Some  show  up  with  their  tools  sharpened 
while  others  benefit  greatly  from  the  aca¬ 
demics  COIs 

•  That  seemed  knowledgeable. 

•  Everyone  was  rusty  because  we  received 
<expletive  deleted>  information  about 
what  we  needed  to  train  on  prior  to  ar¬ 
rival 

•  Some  are  well  prepared 

•  If  anything  the  academics  is  a  refresher 

•  They  are  usually  well  informed 

•  Well  prepared 

•  Have  to  baseline  due  to  varied  back¬ 
grounds.  Some  are  busy  because  they  are 
civilians  for  50  weeks  a  year  but  ramp 
up  quickly 

•  Not  always 

•  I  think  the  staff  are  prepared  in  a  rough 
manner  and  are  waiting  to  find  out  how 
to  apply  what  they  know  to  become  more 
honed 

•  The  academics  were  presented  to  the 
audience.  Interactive  lectures  would  not 
work  in  a  large  group.  There  was  also  a 
concerted  effort  to  respect  the  Multi  na¬ 
tion  audience  and  not  focus  on  US  only. 
Small  teams  (intel)  were  less  formal  and 
more  participatory 


•  Both,  it  depends  on  the  background. 

•  Often.  Academics  don’t  match  the  audi¬ 
ence.  Today  for  example. 

•  NO,  but  that’s  expected,  first  few  days 
are  always  OJT 

•  I  think  most  officers  do,  enlisted  not  so 
much/some 

•  We  are  all  rusty  because  it  is  not  a  part 
of  our  daily  routine  (for  the  most  part). 
As  long  as  no  coherent  joint  strategy  of 
method  of  regular  exposure  to  joint  doc¬ 
trine  exist  this  will  be  the  case. 

•  Varies  with  experience  levels,  attitudes, 
preferences 

•  Academics  were  all  “generalized”  -  very 
little  training  and  PMX.  This  made  ev¬ 
eryone  “rusty”. 

•  This  is  my  first  academics 

•  Those  who  completed  the  online  training 
were  aggravated  they  had  to  sit  through  a 
lot  of  the  same  lectures 

•  Most  seem  to  understand 

•  They  have  had  some  past  job  assign¬ 
ments  that  allowed  them  to  understand 
some  of  the  information  and  reason  for 
the  exercise,  as  a  first  timer  I  did  not 

•  I  believe  most  are  rusty  but  with  practice 
will  be  ready  for  exercises 

•  I  think  most  US  staff  are  well  prepared 

•  Overall  prepared  enough.  Mostly  through 
work/personal  experience  but  maybe  a 
little  rusty  on  current  doctrine 

•  Senior  personnel  know  their  stuff.  Mid 
level  personnel  like  me  don’t  have  that 
advantage. 

•  It  is  hard  to  tell  because  there  is  no  real 
discussion,  but  they  often  forsake  basic 
doctrine  in  daily  life.  Consider  pretesting 
to  opt  out 

•  Depends  on  seniority  and  level  of  joint 
exp.  Varies  widely,  even  at  comm  staff 

•  Very  well 

•  Of  course  they  are  -  send  everyone  to 
MSUC  if  everyone  should  be  on  the 
same  page.  Many  have  no  experience 


on  a  staff,  so  the  ability  to  work  together 
takes  some  time 

•  Somewhat  “rusty”  but  individuals  will  be 
ready  for  the  EX 

In  your  personal  experience,  is 
there  discrepancy  among  trainees’ 
knowledge  and  skill  levels  prior  to 
the  academics?  If  so,  how  signifi¬ 
cant  are  the  differences?  Do  these 
differences  impact  the  exercise? 

•  Yes  huge  individuals  learn  differently 
so  it  is  ridiculous  to  think  CBTs  are  the 
answer 

•  Yes,  very  wide  range  of  knowledge  and 
experience  impacts  the  exercise  in  that 
they  require  more  training  time 

•  No  one  I  have  experienced 

•  Yes,  there  are  discrepancies.  I  do  not 
know  yet  how  they  would  impact  be¬ 
cause  we  did  not  start  yet 

•  Yes,  but  it  all  depends  and  varies. 

•  I  feel  that  the  SMEs  knew  the  material, 
the  issue  I  find  is  their  limited  ability  to 
express  themselves  to  a  mixed  and  gen¬ 
eral  audience 

•  There  may  be  some  insignificant  differ¬ 
ences 

•  Yes,  but  not  significant,  they  have  little 
or  no  impact 

•  Differences  in  knowledge  and  skill  levels 
but  not  significant  enough  to  negatively 
impact  exercise. . .  start  up  current  is  ex¬ 
pected 

•  Yes  can  speed  slow  or  inhibit  the  training 

•  It  may  not  be  effective  to  brief/train  0-6 
down  to  E-4  level.  Also  many  of  the 
partner  nations  have  unknown  skills.  The 
language  barrier  is  formidable  to  effec¬ 
tively  use  all  personnel 

•  Yes,  depends  on  previous  experience,  it’s 
enough  to  carry  the  exercise  but  training 
checklist  for  your  area  (role)  will  assist 
with  expediting  training 

•  Yes,  we  have  people  receiving  training 
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who  will  next  being  performing  func¬ 
tions??? 

•  Yes  but  that’s  why  it  is  an  exercise 

•  Academics  will  usually  only  reawaken 
knowledge  already  learned  rather  than 
instill  new  knowledge.  It’s  a  refresher 
not  a  true  academic  course. 

•  Yes,  not  very,  not  very 

•  This  is  my  first  academics 

•  No,  we  are  always  learning 

•  No  but  there  is  lack  of  understanding 
how  to  bring  it  all  together  in  a  joint  en¬ 
vironment 

•  No,  we  are  always  learning 

•  I  don’t  think  there  are  significant  differ¬ 
ences 

•  Yes,  highly  different  impacts  exercise 
mostly  if  staff  is  not  interested  in  learn¬ 
ing  process 

•  Yes.  There  are  substantial  diffs  (com¬ 
parable  to  diffs  between  HS  grads  and 
PHDs)  which  significantly  impacts  exer¬ 
cises  by  isolating  some  members  while 
others  focus  on  nuance  that  is  irrelevant 
to  the  group 

•  Yes,  minimally,  yes 

•  NO 

In  your  personal  experience,  how 
well  aligned  are  pre-existing  activi¬ 
ties  (e.g.  academics  or  staff  exer¬ 
cise)  to  the  collective  exercise?  For 
instance,  do  the  pre-exercise  train¬ 
ing  and  education  activities  sup¬ 
port  the  collective  event’s  training 
objectives?  Do  the  activities  inten¬ 
tionally  build  on  one  another? 

•  Intent  makes  sense  and  is  helpful  how¬ 
ever  access  (system  down,  CAC  issues, 
etc.)  cause  too  many  issues  and  I  think 
it’s  not  the  best  platform  to  use  solely 

•  They  seemed  to  build 

•  The  academics  and  staff  exercises  are 
vague  and  generic,  and  mostly  oriented 
towards  the  planners.  I  felt  as  though  my 
presence  was  mostly  unnecessary. 

•  Very  well  aligned.  COIs  are  structured 
to  go  from  general  concepts  to  specific 
tasks 

•  I  believe  the  studied  process  to  prepare 
us  was  sufficient. 

•  Not  aligned  at  all 

•  This  go-around  things  seemed  to  be  bet¬ 


ter  prepared  and  aligned 

•  The  pre-training  feels  pointless  when  the 
morning  academics  completely  reiterate 
the  pre-training.  I  am  learning  nothing 
new  so  my  attention  is  not  fully  provided. 

•  Usually  very  well 

•  Well-aligned  support.  Yes,  they  seem  to 
build  on  each  other 

•  Yes/Yes 

•  No  not  always 

•  I  believe  the  pipeline  is  well  planned 

•  Good  timing,  exposure  and  task  density 
for  activities.  The  only  change  is  to  build 
an  event  (2  days)  to  write,  brief,  and  re¬ 
lease  orders 

•  Yes 

•  Not  aligned  or  tailored  to  audience 

•  Too  broad  of  a  question,  basic  answer 
YES 

•  They  support  the  objective,  but  are  not 
useful  as  they  could  be.  Not  valuable/ef¬ 
ficient  use  of  time 

•  Pre-ex  academics  help  bring  everyone  to 
a  common  baseline.  However  more  time 
should  be  taken  to  provide  detailed  CC 
guidance/direction  on  Conops  given  that 
the  originators  of  those  Conops  are  avail¬ 
able.  OPORD  brief,  CC  intent/mission 
brief 

•  We’ll  see 

•  Hard  to  tell 

•  This  is  my  first  academics 

•  Not  fully 

•  Not  very  well  because  all  training  and 
on-line  CBTs  normally  happen  at  your 
own  duty  station  and  not  with  other  ser¬ 
vice  or  in  a  joint  environment 

•  I  believe  they  have  come  a  long  way  and 
will  continue  to  expand/improve  over 
time 

•  I  think  they  are  well  aligned  and  do  build 
on  each  other 

•  They  want  to  achieve  training  objectives 
(not  always  will  happen) 

•  Pre-EX  training  is  a  waste  of  man  hours 
and  resources... few  participate  in  the 
mandatory  training  and  fewer  absorb 
any  INFO.  May  be  better  if  SMEs  taught 
courses. 

•  Could  be  better  tailored  to  various  se¬ 
niority  and  skills  sets  (more  focused  on 
specific  areas  to  enhance  command/di¬ 
rectorate  training  objectives) 


•  Not  well  aligned.  There  was  no  “road  to 
crisis”  orpre  assigned  anything  for  Haiti. 
No  detailed  materials  and  web  maps  of 
everything  weeks  in  advance.  Business 
as  usual  until  “ <expletive  deleted>... 
Haiti  is  underwater!”  The  products  pro¬ 
vided  don’t  mimic  the  products  used 
day-to-day  and  lead  us  to  rely  on  docs 
and  procedures  that  are  not  available  if 
this  contingency  would  happen  in  real 
life. 

•  No  prior  experience  to  drawn  upon 

•  Yes  -  the  planning  process  is  great  for 
those  who  are  able  to  attend 

•  Pre  exercise  training  was  limited 

If  individual  online  courses  could 
be  tailored  to  specifically  support 
pre-training  before  training  events 
such  as  academics,  would  that  be 
helpful?  Why  or  why  not? 

•  Yes  needs  to  be  supplemented  w/  live 
feedback/input 

•  Only  if  it  were  truly  “tailored”  to  each 
person’s  specific  mission 

•  Yes.  Building  them  to  fit  the  supported 
exercise  would  streamline  the  academics 
in  support  of  the  exercise  start 

•  I  believe  it  could  help  and  shorten  the 
academic  training  sessions. 

•  It  would  enable  the  service  members  to 
see  the  entire  picture  of  an  operation  - 
and  possibly  contribute  to  the  concept  as 
“we  train  to  fight!” 

•  Yes,  any  tailored  training  would  assist  in 
better  preparing 

•  Yes,  as  long  as  they  are  both  aligned  to  be 
in  concert  with  each  other  to  build  one’s 
knowledge,  as  of  now  it  is  just  repetitive, 
and  therefore  counter  productive 

•  Yes  that  would  save  time  and  fill  indi¬ 
vidual 

•  Yes 

•  Yes,  reduce  start  up  currency 

•  Maybe 

•  Yes  but  the  scope  is  important 

•  None  come  to  mind 

•  Yes,  as  long  as  it  translates  into  some¬ 
thing  the  member  could  keep  (checklist) 

•  Yes  -  reinforcement.  Maybe  have  online 
training  days  after  academics  as  remind¬ 
er  and  refresher.  USE  online  courses  as 
“prep” 

•  Yes 
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•  Brief  overviews  on  concepts  that  apply 
to  your  function 

•  I  don’t  like  or  believe  in  online  training 

•  Yes  if  specific  individual  capabilities  are 
required  they  should  be  communicated 
ahead  of  time. 

•  No!  Because  of  the  issues  of  access,  con¬ 
nectivity,  passwords,  time,  the  entire  au¬ 
tomated  experience  has  been  negative. 

•  Yes-move  from  “theory”  to  application 

•  I  think  it  would,  so  you  would  have  a  bet¬ 
ter  understanding  of  your  responsibility 

•  Online  classes  were  too  long,  and  if  you 
had  to  stop  in  the  middle  and  restart  days 
later,  you  had  to  go  back  through  materi¬ 
al  to  pass  the  test  at  the  end  of  the  lesson 

•  Yes  tech  skills  require  tech  training,  ba¬ 
sic  skill  -  basic  training 

•  Yes  because  it  makes  it  more  meaning¬ 
ful  and  you  actually  get  credit  and  recog¬ 
nized  for  having  specialized  training  and 
it  won’t  be  looked  at  as  just  another  on¬ 
line  CBT  or  requirement  for  deployment. 

•  Absolutely 

•  I  believe  it  would  be  helpful  to  tailor  the 
training  to  specific  functional  areas.  Not 
all  participants  need  to  learn  joint  fires 
panning  or  joint  targeting,  etc. 

•  Not  really.  Unless  it  uses  past  exercises 
as  framework  example 

•  NO.  eLearning  does  not  address  system¬ 
ic  problems  within  the  military’s  psy¬ 
chology.  Tailoring  courses  would  simply 
become  another  obstacle  to  completing 
an  exercise. 

•  Yes.  Allow  focus  on  specific  functions/ 
warfare  areas 

•  Yes.  You  could  learn  about  what  you 
were  actually  going  to  be  doing. 


•  Not  helpful.  We’re  not  standing  the 
“BML”  we’re  trying  to  stop  them,  up  to 
kinetic  force 

•  Yes.  Focus  on  relevant  topics  might  help 
individuals  care  more  and  not  just  check 
the  block  (box) 

•  No 

If  implemented,  what  topics  should 

online  pre-exercise  courses  focus 

on? 

•  Language,  culture,  maybe  setup  for 
learning  doctrine  items  in  different  for¬ 
mats  (e.g.  crosswords,  matching  etc.). 
This  engages  the  student  at  different  lev¬ 
els. 

•  Products  (general)  that  would  have  to  be 
produced  during  the  exercise 

•  Depending  on  your  function,  provide 
additional  online  pre-exercise  courses, 
i.e.  I’m  assigned  to  C4/J4  as  a  Logistics 
Planner,  as  such  training  in  Joint  Logis¬ 
tics  &  op  planning  would  help 

•  Command  structure,  communications, 
basically  just  what  it  is  now,  just  better 
presented 

•  The  current  array  of  courses  serves  well 

•  Joint  Training  (Doctrine) 

•  Depends  on  the  exercise  scenario.  Partic¬ 
ipants,  area  of  operations,  resources/sup- 
port,  interagency,  IGO,  multi-national 

•  Focus  on  concepts,  terms,  “who  is  who 
in  the  zoo,”  what’s  the  op  flow  chart, 
flow  of  support  to  end  result.  Train  the 
specifics  onsite 

•  Information  flow  of  exercises 

•  Culturally  based  and  focused  on  the  par¬ 
ticipating  nations.  Also  some  info  on 
their  uniform  and  ranks.  How  can  we 


salute  if  we  don’t  know  their  ranks  and 
recognize? 

•  Overall  objectives.  Short  training  ses¬ 
sions.  The  long  and  many  modules  were 
too  much 

•  Joint  Pubs  (JP-3-33  and  JP-2-01) 

•  Interactive  planning  and  execution  exer¬ 
cises! 

•  No  on-line  courses  please  stop  the  insan¬ 
ity 

•  PMX-related  courses  (tailored) 

•  They  should  be  tailored  to  your  specific 
role. 

•  Job  specific 

•  Because  of  “globalization”  it  should  all 
focus  on  language  barriers  of  different 
services  when  it  comes  to  a  joint  effort/ 
training/  and  performance. 

•  Each  specific  Jos  in  a  JTF 

•  Specific  functional  area  courses 

•  Be  MORE  interactive  in  decision-mak¬ 
ing  ,  info  sharing  and  processes  to  get 
CDR  approval  on  actions  to  be  taken. 

•  Focus  on  task  at  hand,  that  will  give  you 
the  information  you  need 

•  Background  info  specific  to  the  train¬ 
ing  scenario,  not  doctrinal  courses  that 
individuals  should  already  know  or  for 
which  info  is  readily  accessible 

•  Joint/Combined  best  practices  for  pro¬ 
cesses  in  specific  functional  areas 

•  A  way  to  effectively  focus  on  using  cur¬ 
rent  products  to  support  the  exercise  is 
made  up  documents  we  see  once  a  year. 

•  No  experience  to  draw  upon  to  provide 
recommendations 

•  Topic  specific  to  the  exercise 


Post-Exercise  Reactions 
Survey  (Experimental 
Group) 

What  activities  did  you  complete 
as  part  of  the  PANAMAX  pre-ex¬ 
ercise  work-up? 

•  JTF 

•  Online  courses,  staff  check-ins 

•  Academics,  online  courses 


Online  courses 

PMX12  academics,  online  course 

None,  I  jumped  from  J34  ROE  working 
group/J34  mission  assurance  WG  to  real 
world.  Throughout  the  entirety  of  PAN¬ 
AMAX 

I  completed  all  online  courses  required 
for  PANAMAX.  I  was  involved  with 
coordination  targeting,  CAP,  STAFFEX 
and  planning  J35  FOUPS  activities 

JKO  courses,  Coliseum  and  DCO  were 
helpful 

Academics,  online  classes 


Academic  Beta  online  course 
Academics,  online  courses 
Online  training  and  web  portal 
Online  courses  only 
Online  courses  only 

Online  courses  as  well  as  some  class¬ 
room  course 

J63  OPS  cell 

Joint  operations  planning,  interorgani- 
zational/multinational  coordination,  and 
interorganization  coordination 
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•  Academics,  online  course 

•  No,  total  surprise  when  I  got  here 

•  Vcat  and  the  three  JKO  courses 

•  Online  courses,  JKO,  and  Southcom,  and 
terminology 

•  Staff  Ex  for  two  days  and  computer  hard 
road  to  crisis 

•  Online  courses 

•  PANAMAX  Academics,  Participated  on 
Ex  CAP  CONOPS  TNG,  Participated  on 
Planning  Conferences  JKO  courses 

•  JKO 

•  Online  courses,  PANAMAX  101,  train¬ 
ing  in  CAT 

•  No  prep  other  than  online  courses  and 
short  academics  session.  Daily  duty  per¬ 
formance  prepared  me  more  than  other 
issues. 

•  PANAMAX  pre-exercise  training 

•  Academics,  pre-crisis  deliberate  plan¬ 
ning,  CAP,  online  courses 

•  Academics,  online  courses,  and  internal 
IO  staff  preparation 

•  Academics,  staff  ex,  and  online  courses 

•  I  did  the  CoL  JIEDDO  &  JFC  200  cours¬ 
es  and  I  did  the  SOUTHCOM  J777  pre- 
PANAMAX  online  training  regarding 
the  Road  to  Crisis 

•  All  required  online  courses 

•  Academics  and  online  courses 

Were  these  activities  valuable;  why 

or  why  not? 

•  Yes,  because  I  learn  new  acronyms  and 
who  is  in  charge  of  certain  forces  and  as¬ 
sets 

•  Not  the  online  courses,  but  the  check  in 
was  vital  to  accountability  of  personnel. 

•  No,  I  had  prior  training  in  the  joint  plan¬ 
ning  process  and  many  others  during 
PMX12  seem  not  to  have  conducted 
training. 

•  No,  I  honestly  cannot  discern  a  correla¬ 
tion  between  the  courses  and  PANA¬ 
MAX. 

•  Yes,  good  run-down  of  interagency  pro¬ 
cesses  needed  to  be  oriented  more  to  job 
specific  for  the  actual  end  user 

•  Yes  for  the  MNFS 

•  Yes  the  academics  and  CAP  were  help- 
fill  in  preparing  the  J35  to  plan  for  cri¬ 
sis  action  OPTs  and  coordinate  with  our 
components 


•  The  JKO  course  didn’t  really  help.  The 
DCO  and  Coliseum  training  were  essen¬ 
tial 

•  Neither  were  specific  to  the  actual  exer¬ 
cise 

•  Review-  refresher  help  remind  me  of 
joint  operational  concepts  but  did  not 
have  a  major  impact  during  PMX12  (I’m 
not  in  J3  or  J5  or  J9) 

•  Yes,  needed  more  depth  on  applications 
available  and  how  to  use  (APAN) 

•  Yes  somewhat-role  training  preferred 

•  Not  a  big  fan  of  online  learning 

•  Yes 

•  The  classroom  course  more  related  to  my 
function  during  PANAMAX 

•  No  didn’t  do  anything 

•  Yes  I  learned  more  about  what  goes  on 
during  the  entire  planning  and  execution 
of  multi-national  forces  mission/exer- 
cises 

•  Yes,  help  me  to  better  understand  the  ex¬ 
ercise  and  other  players  role 

•  No,  took  way  too  long  and  the  vocabu¬ 
lary  just  merged 

•  Vcat  was  ok  if  I  were  deploying  to  the 
area  but  not  for  the  CAC 

•  Somewhat 

•  Yes,  save  situational  awareness  of  the 
operation  and  enemy  situation  as  well  as 
provided  guidance  on  the  tools  available 
for  C2  coordination 

•  Yes-  re-familiarize  myself  with  the  ter¬ 
minology 

•  Yes,  kept  me  focused  on  exercise  objects. 

•  My  position  didn’t  require  coordination 

•  Some  regarding  CAT  processes  and  ex¬ 
pectations 

•  Yes,  but  not  nearly  as  valuable  as  they 
could  have  been.  The  training  would 
have  been  far  more  valuable  if  it  includ¬ 
ed  SPECIFICS  -  especially  organization¬ 
al  structure  for  the  exercise 

•  Valuable  as  it  pertains  to  purpose  and 
situational  awareness 

•  Yes,  helped  get  my  head  in  the  game 

•  Yes,  preparing  me  for  the  exercise  mini¬ 
mized  last  minute  information  gaps  and 
scrambling 

•  Yes,  set  the  framework 

•  I  have  attended  the  former  JFCOMJECC 
course  so  I  have  been  through  the  whole 
MOMP  process,  so  I  didn’t  get  much 


from  the  JFC  100  &  200  courses.  Also, 
I  need  to  be  familiar  with  the  MOMP  as 
the  Chief  of  Exercises  but  I  don’t  prac¬ 
tice  it  on  a  daily  basis 

•  Yes,  new  concepts  and  detailed  informa¬ 
tion  in  regards  to  the  overall  exercise 
were  covered. 

•  Yes,  the  activities  were  valuable.  How¬ 
ever,  they  only  provided  a  taste  of  what 
to  expect 

What  different  and/or  additional 

pre-exercise  activities  would  you 

recommend? 

•  Shorter  version 

•  Maybe  courses  that  actually  bear  mean¬ 
ing  to  my  lowly,  insignificant  pay  grade. 

•  Staff  training  to  better  orient  staff  to  ca¬ 
pabilities  in  other  branches. 

•  Cdiserm  and  DCE  training  is  the  only 
requirement  I  could  see  necessary  for 
PANAMAX 

•  Job  specific  training  per  position  and  J 
code,  and  CAC  relationship  with  other 
entities  within  southcom  alone. 

•  No 

•  For  those  not  in  planning  billets,  please 
made  sure  those  individuals  attend 
JECC-PC  in  Suffolk,  especially  those  at¬ 
tending  OPTs  for  PANAMAX 

•  Online  courses  tied  into  PANAMAX 

•  Having  AARs  from  previous  years  and 
lessons  learned 

•  JPOE  and  CPOE,  mission  analysis  and 
center  of  gravity  presentation  on  CBT 
would  be  useful  for  next  exercise 

•  Review  of  the  many  venues/application 
of  how  information  is  passed  to  and  from 
COCOM/interagency/MNFS/Compo- 
nents 

•  Role  simulation 

•  Live  coursework 

•  One  day  to  go  over  PANAMAX  struc¬ 
ture,  OPORP,  etc.  I  was  too  busy  with 
my  normal  work  load  to  prep  for  PANA¬ 
MAX  until  it  kicks  off 

•  None 

•  Annual  online  refresher  training 

•  None-online.  Classroom  day  or  1/2  day 
on  what  you  want 

•  A  good  portion  of  this  is  OJT,  nut  class¬ 
room  training  may  help  on  the  subjects 

•  Don’t  know 
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•  Specific  training  or  to  the  different  func¬ 
tions  within  the  CAT  and  what  it  brings 
to  the  table 

•  Review  of  command  structure  for  PAN- 
AMAX 

•  More  in  depth  command  and  control  re¬ 
lations  as  well  as  current  operations  pro¬ 
cesses  and  procedures 

•  PAN  AM  AX  101 

•  Nothing  comes  to  mind 

•  Practical  tools  for  running  key  B2C2W- 
GS  (e.g.,  FUPLANS,  JPG  vs.  FUOPS, 
PMB,  etc.) 

•  More  reality-based  instruction  on  coordi¬ 
nating  with  other  services,  organizations, 
and  multinational  forces 

•  No  additional  activities 

•  None 

•  Participation  in  all  planning  conferences 
to  understand  better  the  intent,  objec¬ 
tives,  and  everything  else  in  regards  to 
the  exercise  from  the  beginning  stage. 
Also  to  improve  network/interaction 

•  I  would  like  to  see  rehearsals  and  prod¬ 
ucts 

Did  the  online  courses  help  you 
with  your  duties  during  PANA- 
MAX?  If  yes,  can  you  provide  an 
example?  If  no,  what  would  have 
made  the  courses  more  useful? 

•  No.  The  courses  had  nothing  to  do  with 
me.  It  all  seemed  geared  to  the  “upper- 
crust”.  Gear  some  info  to  the  lower  pay- 
grades. 

•  No,  courses  provided  no  additional  in¬ 
formation  to  individuals  with  fonnal 
planned  training. 

•  No,  they  did  not  pertain  to  any  of  my  per¬ 
sonal  duties. 

•  Yes.  Discussed  organizational  structure, 
processes  and  planning,  good  for  overall 
knowledge,  not  for  intel  duties. 

•  None 

•  No,  the  courses  were  not  helpful  for  those 
involved  in  daily  planning  activities 

•  No,  I  couldn’t  link  the  learning  to  PANA- 
MAX 

•  Not  really-  hard  to  remember  all  the  ac¬ 
ronyms 

•  No 

•  No  not  really  involved  in  planning  at 
night.  It  was  more  interagency  and  multi¬ 


national  force  coordination 

•  Break  it  down  to  smaller  parts  (Mind  ab¬ 
sorption) 

•  Yes,  in  tenns  of  understanding  the  lingo 

•  No  we  ended  up  not  using  any  of  the 
systems  that  were  taught  online.  We  used 
white  boarded  concepts  and  PowerPoint 

•  No,  I  am  unsure  of  and/or  cannot  think  of 
a  use  for  the  information  in  the  courses 
relation  to  my  duty  position 

•  Would’ve  been  more  useful  if  CoL  cov¬ 
ered  my  job  responsibilities 

•  No,  most  of  my  duties  and  responsibili¬ 
ties  were  completed  prior  to  the  exercise 

•  No,  but  it  help  me  understand  the  roles 
and  duties  of  other  players 

•  No.  Finding  the  courses  was  a  needle  in  a 
hay  stack.  Not  user  friendly-think  apple/ 
mac. 

•  No,  the  courses  were  too  long  and  the 
subject  matter  was  too  similar  allowing 
it  to  jumble  together 

•  I’m  not  sure  it  helped.  I’m  relatively  new 
to  PANAMAX,  so  everything  seemed 
new.  Can  not  give  you  a  good  enough 
response 

•  Not  a  great  deal,  but  primarily  because 
I’m  always  familiar  with  the  major  pro¬ 
cess  used  in  Joint  planning  and  inter¬ 
agency/  multinational  coordination 

•  Yes 

•  Yes  and  no 

•  No,  my  duties  as  J8  battle  staff  entail  no 
Joint  planning 

•  No  -  not  much  involvement  by  J8  needed 
in  CAT 

•  Not  really 

•  Yes,  provided  a  baseline  for  planning  vo¬ 
cabulary,  processes  and  pitfalls 

•  They  were  a  good  refresher 

•  No,  I  run/aid/contribute  to  the  White 
Cell,  so  I’m  not  part  of  the  training  audi¬ 
ence 

•  Yes,  especially  with  acronyms 

•  Would  have.  I  was  pulled  out  of  PANA¬ 
MAX  prior  to  Start  Ex 

If  you  could  improve  one  or  two  as¬ 
pects  of  online  training,  so  that  it 
better  prepared  staff  for  exercises 
like  PANAMAX  and  related  op¬ 
erational  duties,  what  would  you 
change  and  why? 


I  just  hated  those  courses.  Pretty  much 
everything  about  them. 

Require  all  staff  elements  to  take  training 
and  include  capability  of  various  staff 
elements. 

The  wording  of  the  questions/courses 
was  not  comprehensible  and  did  not 
make  much  sense. 

None. 

None 

Make  the  training  more  related  to  PANA¬ 
MAX,  include  other  terminology  used  in 
PANAMAX  to  the  training. 

Review  lessons  learned  from  previous 
exercises 

Make  these  efforts  more  applicable 
across  the  J-codes 

Have  a  module  with  an  intel  focus 
Role  play 

The  coursework  was  very  detailed. 
Please  put  some  more  rigor  into  showing 
the  big  picture 

I  am  unsure  as  to  the  intent  of  the  training 
as  it  was  not  part  of  my  duty  responsibili¬ 
ties,  during  the  exercise,  to  coordinate  at 
that  level 

None  of  the  officers  with  key  roles  in  my 
shop  wanted  to  do  the  training.  I  think  it 
should  me  mandatory  training  for  all  key 
leaders  involved  in  planning  and  execu¬ 
tion  of  joint  exercises 

It  is  a  band  width  heavy  and  did  not  work 
from  hotel  connection.  Missing  the  why 
a  lot  of  similarly  sounding  “stuff’  and  no 
relevance  to  me. . . .  The  planner 

Shorten  them  and  separate  some  of  the 
subjects.  Classroom  courses  while  here 
would  be  good  also. 

Length  of  course,  felt  like  a  master  level 
class  that  required  too  much  reading  and 
writing 

I  recommend  a  focus  on  process  training. 
Establish  how  can  we  process  informa¬ 
tion  flow  and  intro  USSouthcom  func¬ 
tional  area  to  process  the  information 

Tell  us  the  publications  from  which  to 
draw  the  information  so  we  can  refer  to 
the  pubs 

Provide  specific  to  scenario  subjects  and 
situations 

COP  courses  were  too  long.  It  became  a 
matter  of  completion  instead  of  learning 

The  planning  course  put  me  to  sleep  -  too 
long  too  boring  -  not  very  applicable  (in 
my  limited  experience) 
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•  Nothing  comes  to  mind 

•  Include  online-communication  training 
(DCO,  chat,  etc.)  Many  minutes  and 
hours  wasted  due  to  peoples’  inability  to 
operate  their  computers  effectively 

•  The  length  of  the  training.  The  JKO  por¬ 
tion  took  a  day,  Academics  was  2  days. 
Would  have  been  nice  to  have  a  schedule 
prior  to  the  requirements,  i.e.,  3  months 
out  so  we  could  block  the  time 

•  I  would  include  a  module  with  a  pre-ex¬ 
ecution  checklist  and  requirements  in  an 
effort  to  improve  daily  duties  and  com¬ 
munication  channels 

•  Am  at  a  loss  on  this  one.  In  my  case,  on¬ 


line  training  is  more  accurately  “online 
reading”  and  online  reading,  like  any 
reading,  is  more  familiarization  than 
training 

Any  other  comments  about  the 
pre-PANAMAX  online  courses  or 
eLearning  on  Joint  Knowledge  On¬ 
line,  in  general? 

•  Enough  said. 

•  None,  good  initiative 

•  Need  a  pre  PANAMAX  list  of  require¬ 
ments.  I  hate  surprises  did  not  know  un¬ 
til  I  arrived  even  then,  had  to  get  a  print 
out  and  a  slide  to  know  what  needed  to 


complete. 

•  Online  courses  were  not  beneficial  to  me 
due  to  the  role  I  play  during  PANAMAX. 
But  I  would  surely  recommend  them  for 
those  that  do  (primarily  senior  leader¬ 
ship). 

•  No 

•  Liked  taking  the  test  up  front  to  obtain 
credit  for  what  you  already  know 

•  The  pre-briefed  “expected  time  to  com¬ 
plete”  the  JFC  courses  was  way  off  what 
it  actually  took.  I  think  if  you  include 
the  log  on  and  surveys,  then  each  course 
took  about  2  hours  each. 

•  No 


Post-Exercise  Reactions 
Survey  (Control  Group) 

What  activities  did  you  complete 

as  part  of  the  PANAMAX  pre-ex¬ 
ercise  work-up? 

•  JTF - 1 0 1  Computer  Based  Training 

•  None-  came  from  JECC 

•  Online  courses 

•  Joint  online  JTF  101,  joint  online  human 
rights  and  SERE  1 0 1 .4 

•  The  courses  at  the  bottom  of  the  list. 

•  Marine  net  i4  courses 

•  None 

•  Primary  Professional  Military  Education 
Blocks  1-4,  infonnation  assurance,  PII, 
NWC  naval  logistics  integration,  NWC 
COA  CONOP 

•  Online  courses 

•  Only  one  executive  level  targeting 
course,  however,  I  am  not  filling  that 
duty  position.  I  would  point  out  I  was 
manifested  on  the  JMD  the  week  before 
the  exercise. 

•  Online  courses 

•  JTF  101,  SERE  level  A 

•  All  of  them,  JTF  and  NKO 

•  None 

•  JKO  courses,  PANAMAX  video/courses 

•  Academics  and  online  course 

•  Academics 

•  Online  academics  CBT’S  and  warm-up 
DCO  sessions  hosted  by  my  J-4  plans 


section. 

•  None 

•  Staff-  Ex,  academics  (online  course) 

•  The  PANAMAX  training  provided  on 
the  portal  address  the  load  to  PANA¬ 
MAX,  FDO,  OPSEC,  use  of  APAV,  reg¬ 
istration,  etc. 

•  Academics,  CAP,  Staff-Ex,  online  CBT 

•  Partial  online  JKO  course,  also  prior 
PANAMAX  experience 

Were  these  activities  valuable;  why 

or  why  not? 

•  Yes  they  were.  However  they  seemed 
more  focused  on  higher  ranking  person¬ 
nel. 

•  Yes  because  it  gave  me  a  perspective  of 
working  in  a  joint/multinational  environ¬ 
ment. 

•  Yes,  it  provided  good  fundamental  infor¬ 
mation  on  joint  task  force  operations. 

•  Yes,  necessary  for  annual  training  and 
foreign  travel. 

•  Yes.  General  awareness 

•  It  was  a  wider  “shot  gun”  blast  of  infor¬ 
mation  about  each  field,  not  very  spe¬ 
cific. 

•  No.  Joint  Targeting  course  did  not  pro¬ 
vide  the  fidelity  necessary  to  actually 
conduct  targeting. 

•  They  were  and  I  believe  I  would  have 
gotten  more  out  of  them  if  I  was  in¬ 
formed  earlier  to  take  them 

•  They  were  largely  a  waste  of  time  for 
“Exercise”  purpose  for  real  world  JTF/ 
deploying  ops.  They  would  be  a  lot  more 


viable. 

•  They  may  be  valuable,  but  once  the 
course  is  completed,  everything  was  just 
learned  and  forgotten. 

•  Yes,  they  were  helpful  since  this  was  my 
first  time  here. 

•  Yes,  it  helped  prepare  me  for  the  exercise 

•  Not  so  much  at  my  level.  It  made  it  a 
little  easier  to  understand  how  infonna¬ 
tion  at  higher  levels  flow,  but  no  so  much 
at  the  AO  level. 

•  Yes-  it  helped  me  understand  common 
relationships  and  responsibilities 

•  Yes  staffEx  was  helpful  as  a  warm  up  to 
address  issues  ahead  of  time  and  to  get 
to  know  our  counterparts  in  the  organiza¬ 
tion  playing  in  the  exercise. 

•  Yes  good  SA 

•  Yes,  as  a  reserve  officer  I  need  the  train 
up. 

•  JKO  was  too  long  and  painful  after  an 
hour  or  two  listening  I  realized  I  didn’t 
retain  anything. 

What  different  and/or  additional 

pre-exercise  activities  would  you 

recommend? 

•  None 

•  Bringing  KM  in  earlier 

•  The  pre-ex  activities  I  would  recommend 
are  simply  drills  for  each  directory. 

•  Having  the  road  to  crisis  exercise  brief  in 
advance  would  have  been  good.  I  was  a 
last  minute  billet  fill  and  did  not  get  this 
in  advance. 
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•  Accessing  the  collaboration  tools  and  us¬ 
ing  them 

•  All  necessary  staff  sectors  were  covered. 

•  There  should  be  time  dedicated  to  build 
teams,  identify  systems,  regard  and  ex¬ 
pectations  in  a  collaborated  environment 
at  the  directorate/shift  level. 

•  Some  sort  of  training  to  resolve  the  large 
problem  with  the  RFI  process.  Many 
RFI’s  routed  were  not  drafted  at  the  level 
they  needed  to  be  routed  into  the  CEN- 
TRIX  system.  RFI  quality  was  a  major 
shortcome! 

•  More  OJT  for  the  position  assigned. 

•  More  information  as  to  what  is  going  on. 

•  Send  CBT  list  our  prior  to  exercise  so 
more  time  can  be  taken  to  learn  them 

•  I  would  recommend  a  JIPOE  online  class 

•  We  should  use  actual  systems  /  computer 
applications  to  be  used  during  exercise 

•  None-  Adequate  training  was  required 
and/or  available 

•  Table  top  on  communications  would  be 
helpful 

•  Same,  more  “CAC  responsibilities” 

•  In  person  training.  There  are  three  full 
days  prior  to  StartEx  in  which  someone 
could  lecture  in  the  CAC 

Did  you  complete  any  relevant  on¬ 
line  courses  prior  to  PANAMAX?  If 
yes,  what  courses  did  you  take?  If 
yes,  were  these  courses  useful  for 
PANAMAX? 

•  13  mods  of  JTF  101 

•  Not  specifically  for  PMX,  but  Jecc  per¬ 
sonnel  are  always  prepared. 

•  Yes,  JTF  101  modules,  human  traffick¬ 
ing/ ATF.  The  course  was  useful 

•  Information  assurance,  human  traffick¬ 
ing  as  a  part  of  our  yearly  General  Mili¬ 
tary  Training. 

•  No 

•  No,  I  didn’t  complete  any. 

•  No 

•  No 

•  No.  Again,  I  was  only  recently  told  I  was 
participating. 

•  None 

•  JTF  101,  SERE  level  A.  No  courses  or 
activated  attended  other  than  the  CAP  for 
situational  awareness.  I  am  an  E-6,  there¬ 


fore  options/funds  are  limited. 

•  All  vary  GMTs.  Too  many  to  remember. 

•  Yes,  NKO  courses 

•  No 

•  Yes,  online  course.  It  was  somewhat  rel¬ 
evant  but  it  should  be  more  focus  on  our 
command  and  AOR. 

•  I  finished  AirCommand  Staff  College 
(ACSC)  this  spring  2012-  it  covers  joint 
planning 

•  No 

•  None 

•  No 

•  PANAMAX  required  training  on  SC  por¬ 
tal.  Very  helpful 

•  Not  aware  any  were  available  to  reserve 
officers.  Website?  OWA? 

•  JKO-  I  started  the  required  3  but  didn’t 
finish  them,  I  didn’t  feel  like  I  was  learn¬ 
ing  and  it  was  very  time  consuming. 

If  you  could  improve  one  or  two  as¬ 
pects  of  online  training,  so  that  it 
better  prepared  staff  for  exercises 
like  PANAMAX  and  related  op¬ 
erational  duties,  what  would  you 
change  and  why? 

•  Have  or  create  that  focuses  on  lower 
ranking  personnel. 

•  Bring  KM  early  and  after. 

•  I  would  possibly  add  a  language  intro 
course  or  at  least  be  provided  access  to  a 
Rosetta  Stone  course 

•  None 

•  Having  a  central  point  to  access  all  the 
necessary  courses 

•  More  notice  about  which  classes  need 
to  be  competed  in  the  weeks  before  “de¬ 
ploying”  to  support  PANAMAX 

•  I  don’t  know 

•  I  would  have  insured  more  lead  time 
and  diversity  where  the  classes  can 
be  accessed.  As  a  marine,  I  go  to 
MARINENET  to  find  classes  for  per¬ 
sonal  Pme.  I  ask  to  create  a  sko  account. 
I  believe  the  sites  are  able  to  link  courses. 

•  There  MUST  be  away  to  train  and  test  in¬ 
ject  personnel  on  JTF  CMD  SOPv,  TTPs, 
and  formats. 

•  Have  a  bit  more  interactive  modules.  The 
classes  are  quite  long  and  boring. 

•  When  building  a  Jmo,  try  to  utilize  peo¬ 


ple  that  have  experience  in  that  field.  I 
have  zero  collections  experience,  but  ap¬ 
parently  that  is  plenty  to  be  a  JTF  collec¬ 
tions  manager  (nights). 

•  Make  the  lessons  more  interactive  rather 
that  test  based.  For  example  the  LA  vlO 
course  is  interactive  and  has  no  test.  The 
student  is  not  focused  on  just  answering 
the  question. 

•  More  interactive  and  easier  to  navigate 
and  learn  areas  where  not  proficient 

•  I  would  add  an  online  JIPOE  with  focus 
of  our  AOR.  The  online  course  was  long 
and  painful  and  at  the  level  of  the  course 
it  should  have  tailored  to  your  position 
E-3  m-E-6  don’t  care  about  strategic 
planning  but  the  0  grade  in  planning  the 
class  would  be  good  but  JIPOE  for  the 
worker  would  be  best. 

•  A  little  complicated  but  maybe-create 
module  position-specific  to  run  through 
scenarios  to  help  understand  how  to  han¬ 
dle  issues  of  that  particular  position. 

•  Nothing  to  add,  I  liked  the  warm-ups  like 
DCO’s 

•  Ensure  that  all  or  most  of  the  players  par¬ 
ticipate  in  the  warm-ups. 

•  Nothing-  The  online  training  was  great- 
A  table  top  exercise  pre  PNMX  would 
be  helpful  to  ensure  all  communication 
socialized. 

•  1.  Shorten  it  2.  Tie  it  to  Southcom  prod¬ 
ucts  (OPORD, CONOR  EXORD) 

Any  other  comments  about  the 

pre-PANAMAX  work-up  or  staff 

training  and  education,  in  general? 

•  I  think  that  some  of  the  staffing  could  be 
beefed  up  such  as  personnel  to  aide  camp 
commandant 

•  It  is  difficult  to  access  these  courses  re¬ 
motely  from  home  with  the  current  com¬ 
mon  access  card  (CAC)  software.  It  rare¬ 
ly  works  from  home  if  at  all.  There  needs 
to  be  improvements  in  this  software  to 
be  able  to  be  used  on  most  legacy  Mi¬ 
crosoft  software  (As  most  people  do  not 
update  their  personal  home  software  or 
hardware  as  much  as  the  military  does). 

•  Ensure  people  establish  accounts  prior 
to  the  exercise  and  go  for  a  spin/dry  run 
around  the  site. 

•  I  am  so  ready  for  PANAMAX  13 

•  Pretty  good  exercise  overall. 

•  Too  many  surveys! 

•  I  am  a  reserve  IMA  STA  attached  to  US 
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southcom  so  I  have  a  little  to  no  contact 
with  the  exercise  prior  to  start  ex.  I  typi¬ 
cally  have  one  or  two  days  prior  to  read 
up  on  exercise 

•  Thank  you. 

•  The  portal  is  inundated  with  information, 
it’s  too  much.  I  think  pre-exercise  train¬ 
ing  could  also  just  convert  a  required 
reading  (OPSUM,  OPORD,  COMP,  etc.) 
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Joint  Mindset  Survey 


This  section  lists  the  aggregate  means  and  standard  deviations  on  the  Joint  Mindset  survey.  All  partici¬ 
pants  (experimental  and  control)  completed  this  survey  at  one  point,  during  the  study. 


Joint  Mindset  Questionnaire  Outcomes 

Key: 

1  =  Weekly  2  =  Monthly  3  =  Yearly  4  =  Every  few  years  5  =  Never 


Questionnaire  Items 

“Consider  how  often  you  generally  experience  each ” 

M 

SD 

1.  Desire  a  leadership  role  in  a  project  or  mission. 

2.53 

1.38 

2.  Receive  formal  instruction  addressing  Joint  principles  and  procedures  (class  or  online). 

1.90 

1.11 

3.  Learn  new  Joint  principles  and  procedures  (through  instruction,  discussion,  or  reading). 

2.17 

1.11 

4.  Receive  formal  instruction  addressing  Joint  values  and  ethics  (class  or  online). 

1.71 

1.01 

5.  Outline  specific  ways  that  an  OPLAN  or  CONPLAN  affects  other  US  Services/agencies. 

1.78 

1.27 

6.  Outline  specific  ways  that  an  OPLAN  or  CONPLAN  affects  others  outside  the  US. 

1.73 

1.37 

7.  Seek  out  information  about  activities  and  missions  occurring  outside  your  own  unit. 

2.41 

1.47 

8.  Use  strategic  documents  (e.g.,  National  Security  Strategy)  to  make  decisions. 

2.07 

1.41 

9.  Use  Joint  pubs  and  handbooks  to  learn  and  practice  Joint  terminology. 

2.38 

1.30 

10.  Use  Joint  pubs  to  learn  and  practice  OPORDs,  CONPLANs,  and  OPLANs. 

2.03 

1.33 

11.  Use  Joint  pubs  to  learn  and  practice  Deliberate  Planning  and  Crisis  Action  Planning. 

2.03 

1.38 

12.  Use  own-Service  specific  language  or  slang  around  coworkers  from  other  Services. 

2.93 

1.33 

13.  Learn  about  and/or  discuss  other  Services’ cultures. 

2.94 

1.23 

14.  Learn  about  recent/historic  military  operations. 

2.32 

1.24 

15.  Use  suggestions  from  personnel  in  other  branches  when  creating  orders  and  plans. 

2.50 

1.28 

16.  Have  a  different  perspective  than  personnel  in  your  Service  when  creating  orders/plans. 

2.37 

1.37 

17.  Have  a  different  interpretation  than  personnel  in  your  Service  of  guiding  materials/docs. 

2.38 

1.34 

18.  Do  your  coworkers  at  your  current  duty  assignment  generally  exhibit  Jointness?  (No  =1,  Yes=2) 

1.84 

0.37 

19.  Do  you  have  a  different  interpretation  than  coworkers  of  Joint  values  and  ethics? 

2.12 

1.54 

20.  Do  you  have  a  different  interpretation  than  coworkers  of  Joint  policies  and  procedures? 

2.12 

1.49 
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Stakeholder  Knowledge  Elicitation 
Semi-Structure  Interview 


This  section  lists  the  initial  interview  script  used  for  the  semi-structured  stakeholder  interviews,  which 
were  conducted  during  the  front-end  analysis.  This  script  helped  initiate  the  conversation,  but  then 
stakeholders  were  encouraged  to  provide  additional  comments  throughout  the  discussion. 


1 .  How  many  online  courses  does  (your  organization,  e.g.,  TRADOC)  currently  support? 

2.  What  major  obstacles  do  your  personnel  usually  experience  with  online  training?  Can  you  give  us  some  examples? 

3.  How  are  your  online  courses  aligned  with  field  manuals  and  future  events,  such  as  upcoming  training  events  or  specific 
operational  activities?  Can  you  give  us  some  examples? 

4.  What  percentage  of  the  courses  are  aligned  in  this  manner  (versus  developmental  courses,  like  the  action  officer  develop¬ 
ment  course,  or  compliance-oriented  courses,  like  alcohol  and  drug  awareness)? 

5.  How  do  you  measure  for  knowledge  transfer  in  your  courses;  do  you  assess  higher-levels  of  cognition?  Can  you  give  us 
some  examples? 

6.  What’s  your  sense  of  personnel’s  general  outlook  towards  online  training?  Does  the  culture  seem  to  treat  it  as  an  addi¬ 
tional  burden  or  as  a  value-added  instructional  opportunity?  Can  you  give  us  some  examples? 
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